N 

NAS 

1  R 

NAE 

1  c 

lOM 

Human  Factors  Aspects 
of  Simulation 


rV-'s'documert  has  been  approved 
t  ■  p’  blic  release  and  sale;  its 
| _ /ti'.mbution  is  unlimited. _ _ — 


HI 


C 

|ELECT£ 
k  OCT  7  886 


i  ^  ua  09a 


SECURITY  CLASSIFICATION  OF  THIS  PAGE  fTfflwl  Data  EnlaraF; 


REPORT  DOCUMENTATION  PAGE 


■(font  number 


READ  INSTRUCTIONS 
BEFORE  COMPLETING  FORM 


4  title  (md  Sum il.) 

Human  Factors  Aspects  of  Simulation: 
Report  of  the  Working  Group  on  Simulation 


7.  AuThOAi '•! 

Edward  R.  Jones,  Robert  T.  Hennessy,  Stanley 
Deutsch 


S  TYPE  OF  REPORT  A  PERIOD  COVERED 


Technical  Report 


•  PERFORMING  ORG.  REPORT  NUMBER 


».  CONTRACT  OR  GRANT  NUMBEAr«J 


N00014-85-G-0093 


PROGRAM  ELEMENT.  PROJECT.  TASK 
AREA  4  WORK  UNIT  NUMBERS 


t.  PERFORMING  ORGANIZATION  NAME  ANO  AODRESS 

Committee  on  Human  Factors 
National  Research  Council 
Washington,  D.C.  20418 


II  CONTROLLING  OFFICE  NAME  ANO  AOORESS 

Engineering  Psychology  Program 
Office  of  Naval  Research 
Arlington.  Virginia  22217 


MONITORING  AGENCY  NAME  A  ADDRESS^/  dtHtrmnt  team  Cantnlllni  Otllcd)  IS.  SECURITY  CLASS.  Col  Fill  rtpart) 


Work  Unit  No.  NR196-167 


IZ  report  DATE 

September,  1985 


is  NUMBER  OF  PAGES 


Unclassified 


<Sa  DECLASSIFICATION'  DOWNGRADING 
SCHEDULE 


i«  Distribution  statement  rai  thi,  ftapan; 


Approved  for  Public  Release:  Distribution  Unlimited 


t7.  DISTRIBUTION  STATEMENT  (of  ih«  obotroct  mntorod  In  Block  70,  II  dllloront  from  Report) 


i».  supplementary  notes 

The  Committee  on  Human  Factors  is  sponsored  by  the  Army  Research  Institute, 
the  Office  of  Naval  Research,  the  Air  Force  Office  of  Scientific  Research, 
the  National  Science  Foundation,  and  the  National  Aeronautics  and  Space 
Administration 


19-  KEY  WORDS  f  Confirm*  on  teeoree  old*  II  nocooeocy  and  Identity  by  block  number) 

simulators  Committee  on  Human  Factors 

simulation  human-machine  interaction 

engineering  design  simulator  research 

training 


20.  ABSTRACT  (Continue  on  eoeoroo  cl  do  II  nocoeoory  md  Identity  by  block  mmbor) 

The  increasing  importance  and  pervasiveness  of  simulation  are  due  to  several 
factors  including  cost  and  time  savings  that  can  be  realized  in  system  design 
and  training,  the  ability  to  reproduce  and  examine  situations  that  would  be 
unsafe  using  actual  equipment,  the  control  and  measurement  of  human-machine 
performance  in  a  benign  environment,  and  the  capability  to  investigate  con¬ 
ditions  that  would  be  impractical  to  arrange  otherwise.  Shortcomings  in  the 
use  of  current  simulators  are  described  and  recommendations  are  made  for 

and  use  of  simulators.  X _ _ 


FmTWM  iVTh7  CT  ■vKil  iFTiwMr  Pa 


00  ,j2Tt,  U7J 


EDITION  OF  I  NOV  AS  It  OBSOLETE 

S  N  0J0MF- 014- 4401 


IECUNiTT  CLASSIFICATION  OF  TNI*  BABE  fOAwi  Bum 


Human  Factors  Aspects 
of  Simulation 


Edward  R.  Jones,  Robert  T.  Hennessy, 
and  Stanley  Deutsch,  Editors 


Working  Group  on  Simulation 
Committee  on  Human  Factors 

Commission  on  Behavioral  and  Social  Sciences  and  Education 


National  Research  Council 


Sc 


DT1C 

,ELECTEI 


7  885 


\  y  <  S'  *  < ,  \ 

VW  ■>> 


NATIONAL  ACADEMY  PRESS 
Washington.  D.C.  1985 


I 


NOTICE:  The  project  that  is  the  subject  of  this  report  was 
approved  by  the  Governing  Board  of  the  National  Research 
Council,  whose  members  are  drawn  from  the  councils  of  the 
National  Academy  of  Sciences,  the  National  Academy  of 
Engineering,  and  the  Institute  of  Medicine.  The  members  of  the 
committee  responsible  for  the  report  were  chosen  for  their 
special  competences  and  with  regard  for  appropriate  balance. 

This  report  has  been  reviewed  by  a  group  other  than  the 
authors  according  to  procedures  approved  by  a  Report  Review 
Committee  consisting  of  members  of  the  National  Academy  of 
Sciences,  the  National  Academy  of  Engineering,  and  the  Institute 
of  Medicine. 

The  National  Research  Council  was  established  by  the  National 
Academy  of  Sciences  in  1916  to  associate  the  broad  community  of 
science  and  technology  with  the  Academy's  purposes  of  furthering 
knowledge  and  of  advising  the  federal  government.  The  Council 
operates  in  accordance  with  general  policies  determined  by  the 
Academy  under  the  authority  of  its  congressional  charter  of 
1863,  which  establishes  the  Academy  as  a  private,  nonprofit, 
self-governing  membership  corporation.  The  Council  has  become 
the  principal  operating  agency  of  both  the  National  Academy  of 
Sciences  and  the  National  Academy  of  Engineering  in  the  conduct 
of  their  services  to  the  government,  the  public,  and  the 
scientific  and  engineering  communities.  It  is  administered 
jointly  by  both  Academies  and  the  Institute  of  Medicine.  The 
National  Academy  of  Engineering  and  the  Institute  of  Medicine 
were  established  in  1964  and  1970,  respectively,  under  the 
charter  of  the  National  Academy  of  Sciences. 

The  Committee  on  Human  Factors,  in  the  Commission  on  Behavioral 
and  Social  Sciences  and  Education,  is  sponsored  jointly  by  the 
Air  Force  Office  of  Scientific  Research,  the  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences,  the  Office  of 
Naval  Research,  the  National  Aeronautics  and  Space 
Administration,  and  the  National  Science  Foundation. 

This  work  relates  to  Department  of  the  Navy  Grant 
N00014-85-G-0093  issued  by  the  Office  of  Naval  Research  under 
work  unit  no.  NR  196-167.  However,  the  content  does  not 
necessarily  reflect  the  position  or  the  policy  of  the 
government,  and  no  official  endorsement  should  be  inferred. 

The  United  States  government  has  at  least  a  royalty-free, 
non-exclusive  and  irrevocable  license  throughout  the  world  for 
government  purposes  to  publish,  translate,  reproduce,  deliver, 
perform,  dispose  of,  and  to  authorize  others  so  to  do,  all  or 
any  portion  of  this  work. 


Available  from 
Committee  on  Human  Factors 
National  Research  Council 
2101  Constitution  Avenue 
Washington,  D.C.  20418 


l 


WORKING  GROUP  ON  SIMULATION 


EDWARD  R.  JONES  (Chair) ,  McDonnell  Douglas  Corporation, 
St.  Louis,  Mo. 

SHELDON  BARON,  Bolt  Beranek  and  Newman  Inc.,  Cambridge, 
Mass. 

PAUL  W.  CARO,  Seville  Research  Corporation,  Pensacola, 
Fla. 

JAMES  G.  GREENO,  Department  of  Psychology,  University  of 
Pittsburgh 

RICHARD  S.  JENSEN,  Department  of  Aviation  Psychology, 

Ohio  State  University 

HERSCHEL  W.  LEIBOWITZ,  Department  of  Psychology, 
Pennsylvania  State  University, 

JOHN  A.  MODRICK,  Honeywell,  Inc.,  Minneapolis,  Minn. 

JESSE  ORLANSKY,  Science  and  Technology  Division, 

Institute  for  Defense  Analyses,  Alexandria,  Va. 

JOHN  B.  SINACORI ,  John  B.  Sinacori  Associates,  Hollister, 
Calif. 

DONALD  VREULS,  Vreuls  Research  Corp.,  Thousand  Oaks, 
Calif. 

ROBERT  C.  WILLIGES ,  Department  of  Industrial  Engineering 
and  Operations  Research,  Virginia  Polytechnic 


COMMITTEE  ON  HUMAN  FACTORS 


THOMAS  B.  SHERIDAN  (Chair),  Department  of  Mechanical 
Engineering,  Massachusetts  Institute  of  Technology 

nancy  S.  ANDERSON,  Department  of  Psychology,  University 
of  Maryland 

ALPHONSE  CHAPANIS,  Alphonse  Chapanis,  Ph.D.,  P.A., 
Baltimore,  Md. 

JEROME  ELKIND,  Systems  Development  Department,  Xerox 
Corporation,  Palo  Alto,  Calif. 

BARUCH  FISCHHOFF,  Decision  Research  (a  branch  of 
Perceptronics,  Inc.),  Eugene,  Ore. 

OSCAR  GRUSKY,  Department  of  Sociology,  University  of 
California,  Los  Angeles 

ROBERT  M.  GUION,  Department  of  Psychology,  Bowling  Green 
State  University 

JULIAN  HOCHBERG,  Department  of  Psychology,  Columbia 
University 

K.H.  EBERHARD  KROEMER,  Ergonomics  Laboratory,  Virginia 
Polytechnic  Institute  and  State  University 

THOMAS  K.  LANDAUER,  Bell  Communications  Research,  Murray 
Hill,  N.J. 

JUDITH  REITMAN  OLSON,  Graduate  School  of  Business 
Administration,  University  of  Michigan 

RICHARD  w.  PEW,  Experimental  Psychology  Department,  Bolt 
Beranek  and  Newman  Laboratories,  Cambridge,  Mass. 

STOVER  SNOOK,  Ergonomics,  Liberty  Mutual  Research  Center, 
Hopk inton.  Mass. 

ROBERT  C.  WILLIGES ,  Department  of  Industrial  Engineering 
and  Operations  Research,  Virginia  Polytechnic 
Institute  and  State  University 

STANLEY  DEUTSCH,  Study  Director 

MARGARET  A.  CHENG,  Administrative  Secretary 


v 


FOREWORD 


The  Committee  on  Human  Factors  was  established  in 
October  1980  by  the  National  Research  Council.  The 
committee  sponsors  are  the  Office  of  Naval  Research,  the 
Air  Force  Office  of  Scientific  Research,  the  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences,  the 
National  Aeronautics  and  Space  Administration,  and  the 
National  Science  Foundation.  The  principal  objectives  of 
the  committee  are  to  provide  new  perspectives  on 
theoretical  and  methodological  issues,  identify  basic 
research  needed  to  expand  and  strengthen  the  scientific 
basis  of  human  factors,  and  to  attract  scientists  both 
inside  and  outside  the  field  to  perform  the  needed 
research.  The  goal  of  the  committee  is  to  provide  a 
solid  foundation  of  research  as  a  base  on  which  effective 
human  factors  practices  can  build. 

Human  factors  issues  arise  in  every  domain  in  which 
humans  interact  with  the  products  of  a  technological 
society.  In  order  for  the  committee  to  perform  its  role 
effectively,  it  draws  on  experts  from  a  wide  range  of 
scientific  and  engineering  disciplines.  The  committee 
includes  specialists  in  the  fields  of  psychology, 
engineering,  biomechanics,  cognitive  sciences,  machine 
intelligence,  computer  sciences,  sociology,  and  human 
factors  engineering.  Other  disciplines  participate  in 
the  working  groups,  workshops,  and  symposia  sponsored  by 
the  committee.  Each  of  these  disciplines  contributes  to 
the  basic  data,  theory,  and  methods  required  to  improve 
the  scientific  basis  of  human  factors. 


PREFACE 


The  Committee  on  Human  Factors  established  the  working 
Group  on  Simulation  for  the  purpose  of  identifying  behav¬ 
ioral  issues  and  problems  common  to  the  wide  variety  of 
uses  and  types  of  simulation  involving  human  participants. 
The  objectives  of  the  study  were  to  assess  broadly  the 
design  and  use  of  simulation  and  to  recommend  behavioral 
research  and  other  courses  of  action  that  will  improve 
their  effectiveness.  The  field  of  simulation  is  of 
increasing  national  interest  and  of  particular  importance 
to  most  of  the  committee's  sponsoring  organizations, 
which  are  heavily  involved  in  system  development  and 
training  and  need  to  ensure  system  readiness  and 
operational  utility.  Simulators,  depending  on  their 
purpose  and  the  systems  represented,  are  found  in  various 
degrees  of  complexity  and  in  an  increasing  variety  of 
settings  in  addition  to  the  military  and  other  government 
agencies,  such  as  commercial  airlines,  universities,  and 
the  aerospace  and  nuclear  electric  power  industries. 

During  the  period  from  January  1982  to  July  1983,  the 
working  group  members  examined  the  entire  field  of 
simulation.  We  were  able  to  assess  the  fragmented 
activities  in  a  cohesive  way  and  gained  some  interesting 
insights  on  needs  for  behavioral  research,  methodology 
development,  the  use  of  simulation  for  research  and 
systems  acquisition,  the  use  of  simulation  for  initial 
and  operational  training,  and  education  and  training 
needs  for  developers  and  users.  The  resulting  report  is 
an  overview  of  simulation  and  a  guide  for  fundamental 
research  and  educational  activities  that  can  improve 
future  design  and  use  of  simulation.  It  is  intended  for 
the  use  of  groups  concerned  with  simulation,  both  inside 
and  outside  the  human  factors  community,  including  those 
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involved  in  research,  training,  engineering,  systems 

acquisition,  operations,  programming,  and  budgeting.  i 

It  is  important  to  note  that  members  of  the  group  were 
familiar  with  many  of  the  major  activities  that  have  not 
been  reported  in  the  open  literature  in  widely  available 
scientific  and  engineering  journals  or,  in  many  cases, 
not  reported  at  all,  as  is  typical  of  much  work  accom¬ 
plished  in  industrial  or  operational  settings  and  of 
information  that  resides  in  military  documents  with 
limited  distribution.  The  majority  of  this  literature 
was  found  to  be  highly  specific  to  particular  simulators, 
predominantly  training  simulators,  and  describes  physical 
characteristics,  evaluation  of  performance,  or  recommenda¬ 
tions  for  research  within  a  limited  scope.  Most  simula¬ 
tion  research  reports  consist  of  comparisons  among 
specific  alternatives  rather  than  tests  of  general 
principles. 

In  addition,  a  great  deal  of  information  was  obtained 
by  various  working  group  members  at  conferences  and 
workshops  held  by  the  Department  of  Defense  and  the 
National  Aeronautics  and  Space  Administration,  visits  to 
facilities  and  program  offices  such  as  the  Navy  Visual 
Technology  Research  Simulator  in  Orlando,  Florida,  and 
the  office  of  the  Air  Force  Deputy  for  Simulators  at 
Wright-Patterson  Air  Force  Base,  Ohio. 

I  thank  the  members  of  the  working  group  for  the  many 
hours  they  devoted  to  the  meetings  and  to  preparing 
materials  for  the  report.  They  were  especially  adept  at 
accommodating  a  variety  of  viewpoints.  I  am  grateful  to 
the  Committee  on  Human  Factors,  whose  important  contribu¬ 
tions  included  determination  of  the  need  for  the  study 
and  providing  an  overview. 

I  am  grateful  to  Robert  T.  Hennessy,  who  helped  to 
organize  the  working  group  meetings,  for  his  active 
participation  in  the  meetings  and  substantial  contribution 
to  the  content  of  this  report.  In  addition,  he  performed 
the  unenviable  editorial  task  of  merging  the  written 
pieces  contributed  by  each  of  the  working  group  members 
into  a  coherent  report.  His  efficient  handling  of 
administrative  matters  of  the  working  group  is  also 
gratefully  acknowledged. 

I  appreciate  the  efforts  of  Stanley  Deutsch,  study 
director,  for  his  extensive  assistance  in  the  preparation, 
editing,  and  organization  of  the  final  report  for 
publication. 

Jeanne  Richards,  Anne  Sprague ,  and  Margaret  Cheng, 
administrative  secretaries  for  the  Committee  on  Human 
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Factors,  deserve  special  thanks  for  the  multiple  duties 
they  performed  so  well  in  support  of  the  working  group  by 
arranging  meetings,  typing  drafts,  and  distributing 
written  materials  and  information  to  the  members. 
Substantial  editorial  improvements  to  the  economy, 
sharpness  of  expression,  and  organization  of  the  report 
were  made  by  Christine  McShane,  editor  of  the  Commission 
on  Behavioral  and  Social  Sciences  and  Education.  I  thank 
her  for  helping  both  the  writers  and  the  readers  of  this 
report. 


Edward  R.  Jones,  Chair 
Working  Group  on  Simulation 
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SUMMARY 


The  increasing  importance  and  pervasiveness  of  simula¬ 
tion  is  due  to  a  variety  of  factors ,  including  cost  and 
time  savings  that  can  be  realized  in  system  design  and  in 
training,  the  ability  to  reproduce  and  examine  situations 
that  would  be  unsafe  using  actual  equipment,  the  control 
and  measurement  of  human-machine  performance,  and  the 
capability  to  investigate  conditions  that  would  be  imprac¬ 
tical  to  arrange  otherwise.  Additional  impetus  has  been 
provided  by  the  tremendous  technological  advances  in 
computer  hardware  and  software  capabilities,  especially 
in  visual  simulation,  allowing  for  the  creation  and 
control  of  complex,  realistic  environments. 

Simulation  is  used  to  support  the  design,  development, 
and  test  of  many  advanced  systems  and  to  provide  for 
training  on  their  operations;  for  example,  it  enables  an 
aircraft  to  "fly"  realistic  missions  years  before  its 
first  actual  flight.  The  technology  incorporated  into  a 
new  system  may  have  been  developed  using  research  simu¬ 
lators,  and  the  test  and  evaluation  of  the  integrated 
system  may  have  been  accomplished  largely  using  engineer¬ 
ing  simulators. 

The  Working  Group  on  Simulation  was  asked  to  assess 
broadly  the  design  and  use  of  simulators,  to  identify 
behavioral  issues  and  problems  common  to  different  uses 
and  types  of  simulators,  and  to  recommend  fundamental 
research  and  other  courses  of  action  to  improve  the 
understanding  and  efficiency  of  human  factors  applications 
in  simulation.  The  range  of  applications  they  considered 
includes  training,  system  design  and  development,  test 
and  evaluation,  and  research.  The  emphasis  was  primarily 
on  the  use  of  simulators  by  the  Department  of  Defense 
(DoD)  and  the  National  Aeronautics  and  Space  Administra¬ 
tion  (NASA)  as  major  users  of  simulators;  however,  other 


1 


users  were  also  considered.  The  working  group  also 
considered  the  changing  role  of  the  human  as  an  operator 
and  maintainer  resulting  from  increased  automation,  and 
the  effect  of  the  resulting  complex  decision-making  roles. 
The  uses  of  simulation  and  the  relevant  literature  were 
examined  systematically. 

The  report  identifies  behavioral  issues  and  problems 
common  to  many  simulators  involving  human  participation 
and  recommends  research  intended  to  enhance  their  effec¬ 
tive  use.  The  report  contains  an  overview  of  simulation 
including  some  historical  perspective  and  provides  a 
guide  for  research  and  educational  practices  aimed  at 
improving  the  future  design  and  use  of  simulators. 

The  working  group  represented  a  broadly  balanced 
composition  of  disciplines  and  organizations  as  shown  on 
page  iii  of  this  report.  Their  areas  of  specialization 
include  systems  and  design  engineering,  simulation 
engineering  and  training,  control  systems,  performance 
measurement,  psychology  and  cost-benefits  analysts.  They 
represent  expertise  in  industry,  university,  and  govern¬ 
ment  R&D  and  operational  organizations  (see  Appendix  B 
for  biographical  sketches  of  the  working  group  members) . 


RESEARCH  ISSUES 

The  working  group  identified  nine  specific  issues 
involved  in  the  design,  use,  and  application  of  simu¬ 
lators;  most  of  them  have  to  do  with  features  and 
practices  for  effective  employment  of  simulators.  They 
are  examined  in  detail  in  Chapter  5.  These  nine  major 
research  issues  were  organized  into  three  major  cate¬ 
gories:  (1)  Design — training  simulator  design  guides, 

improved  task  analyses  methodology,  and  stress  in  simu¬ 
lator  training  and  testing;  (2)  Operational — scenario 
construction  and  simulation  control  support  features, 
experimental  design,  and  performance  measurement;  and  (3) 
Applications — training  simulator  instruction  guides, 
cognitive  skills,  and  training  for  failures  and 
out-of-tolerance  conditions. 

Behavioral  issues  are  of  particular  concern  in  the 
development  of  simulators,  and  major  improvements  in 
simulation  involve  behavioral  science  research  as  well  as 
technological  advances.  One  concern  is  that  human 
factors  and  equipment  issues  tend  to  be  viewed  separately 
when  in  fact  they  are  intimately  intertwined  in  terms  of 
system  design  and  testing,  personnel,  selection,  and 
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operational  training.  Rather  than  being  isolated,  these 
areas  should  be  developed  through  a  system  engineering 
process  that  fully  integrates  human  factors  engineering 
considerations.  The  concern  is  how  functional  require¬ 
ments  become  defined  and  translated  into  physical  simu¬ 
lator  characteristics,  and  how  the  devices  are  acquired 
and  used.  The  process  has  a  major  impact  on  equipment 
and  personnel  costs  and  on  the  effectiveness  of  the  end 
product.  Although  procurement  and  design  practices 
continue  to  concentrate  on  the  physical  characteristics 
of  simulators  for  systems  design  and  simulators  for 
training,  psychological  factors  (i.e.,  the  behavioral 
objectives)  should  be  given  greater  consideration  in 
simulator  design  and  in  procurement  specifications. 


CONCLUSIONS  AND  RECOMMENDATIONS 

In  discussing  the  problems  involved  in  simulation,  the 
group  arrived  at  eight  interrelated  conclusions: 

1.  Physical  correspondence  of  simulation  is  overemphasized 
for  many  purposes,  especially  training. 

2.  Simulators  are  often  not  used  properly. 

3.  Simulation  could  be  more  cost-effective. 

4.  The  role  of  behavioral  science  and  human  factors 
engineering  in  simulation  is  neglected. 

5.  Many  persistent  simulation  problems  are  common  across 
types  and  uses. 

6.  Our  capability  to  measure  operator  performance  is 
limited. 

7.  The  use  of  modeling  in  simulation  is  not  well 
developed . 

8.  Science  and  technology  of  simulation  are  not  well 
developed  or  integrated. 

On  the  basis  of  discussions  of  these  fundamental 
problems,  the  working  group  makes  three  key  recommenda¬ 
tions*  to  create  the  conditions  essential  to  attack  them: 


♦Working  group  member  Jesse  Orlansky  disagrees  with  the 
first  two  recommendations;  his  dissent  appears  at  the  end 
of  Chapter  7. 
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1.  Long-range,  comprehensive,  and  forward-looking  research 
plans  should  be  developed  to  address  persistent  and 
emerging  simulation  problems. 

2.  Long-range,  stable  funding  should  be  provided  to 
encourage  the  development  of  academic  bases  for 
simulation  research. 

3.  Research  to  develop  near-real-time  human  performance 
assessment  capability  for  simulation  is  urgently 
needed. 


ORGANIZATION  OF  THE  REPORT 

Chapter  1  provides  background  information  on  the 
characteristics  of  simulation.  Chapter  2  focuses  on  how 
simulators  have  been  used  and  issues  in  their  applica¬ 
tions.  Chapter  3  identifies  many  of  the  behavioral 
issues  that  apply  to  the  design  of  simulators.  Chapter  4 
describes  earlier  contributions  of  behavioral  sciences  to 
simulation.  Chapter  5  describes  specific  behavioral 
research  issues  relevant  to  simulation.  Chapter  6 
predicts  future  trends  in  systems  and  simulation.  Itie 
final  chapter  summarizes  the  findings  and  makes  a  set  of 
three  key  recommendations.  The  report  has  two  appendixes: 
Appendix  A  provides  examples  of  particular  simulators; 
Appendix  B  contains  biographical  sketches  of  the  working 
group  members. 


1 


INTRODUCTION 


Although  the  terra  simulation  has  multiple  meanings  that 
differ  according  to  various  classes  of  applications,  this 
report  refers  only  to  simulation  that  involves  human 
interaction  with  equipment  and  that  permits  observation 
or  analysis  of  analogues  of  real-world  situations  not 
otherwise  accessible  with  adequate  convenience,  unobtru¬ 
siveness,  cost  savings,  or  safety.  Simulation  in  the 
context  of  this  report  is  the  representation  of  equipment, 
systems,  events,  and  interaction  processes.  A  simulator 
may  be  similar  to  the  real  equipment  in  some  ways  and 
unlike  it  in  others;  some  components,  functions,  or  system 
events  may  not  be  represented  at  all.  The  realism  and 
comprehensiveness  of  a  particular  simulation  need  only  be 
sufficient  for  the  user's  purposes;  trade-off  decisions 
are  made  to  determine  the  extent  of  realism  and  compre- 
hensivenesss  that  is  necessary  and  affordable.  Factors 
such  as  the  objectives  of  the  designer  or  user,  the 
questions  the  user  wishes  to  be  answered,  data  to  be 
provided,  analysis  to  be  performed,  and  the  behavior  to 
be  elicited  determine  what  must  be  represented  in  the 
simulation.  The  medium  for  implementing  the  simulation 
(digital  computer,  electro-mechanical,  or  manual,  for 
example)  may  also  determine  the  form  of  the  simulation. 

Simulators  are  used  for  four  fundamental  purposes:  (1) 
training;  (2)  systems  and  equipment  design,  development, 
test,  and  evaluation;  (3)  research  on  human  performance; 
and  (4)  licensing  and  certification.  However,  this 
simple  division  is  not  absolute.  For  example,  battle  or 
warfare  simulators  are  used  frequently  to  develop  tactics, 
train  participants  in  combat  management  skills,  and 
evaluate  the  operational  system.  Thus,  the  use  may  be  a 
blend  of  more  than  one  purpose — in  the  case  of  battle 


5 


6 


simulators,  systems  development,  operational  procedures, 
and  training. 


CHARACTERISTICS  OF  SIMULATORS 

Simulators  differ  in  their  physical  characteristics. 

The  most  prominent  physical  dimensions  that  distinguish 
them  are  their  realism  and  comprehensiveness.  By  realism 
is  meant  the  fidelity  of  physical  representation.  A 
simulator  may  be  very  abstract,  consisting  of  a  computer 
representation  of  a  process  displayed  on  a  screen  con¬ 
trolled  through  a  keyboard.  At  the  other  extreme,  the 
simulator  may  consist  of  the  actual  equipment  augmented 
with  additional  equipment  or  special  software.  Most 
simulators  fall  between  these  extremes  of  realism. 

Independent  of  the  degree  of  realism,  simulators  may 
also  differ  along  the  dimension  of  comprehensiveness. 
Comprehensiveness  is  the  degree  of  completeness  and 
accuracy  of  representation  of  all  functions,  environmental 
characteristics,  situational  factors,  and  external  events 
that  are  present  in  the  target  system  or  affect  its  func¬ 
tion.  Fidelity  of  simulation,  usually  intended  to  mean 
some  combination  of  realism  and  comprehensiveness,  is  a 
topic  of  much  concern  and  controversy  that  is  discussed 
at  length  in  Chapter  3. 

Simulators  generally  have  scenarios — that  is,  a 
schedule  or  script  that  drives  the  course  of  events  and 
action  of  simulated  components  to  which  the  person  or 
system  under  study  reacts.  Commonly,  some  computational 
rules  and  data  files  are  provided  for  the  purpose  of 
generating  the  sequence  of  scenario  events;  they  may  also 
be  used  in  keeping  track  of  events.  In  some  cases,  such 
as  air  combat  involving  live  participants,  human  initia¬ 
tive  determines  the  course  of  events.  In  many  cases  the 
actions  of  simulated  agents  as  adversaries  or  other  team 
members  provide  the  cues  or  conditions  for  responses  from 
the  participant.  The  sequence  may  be  deterministic  or 
probabilistic.  In  a  deterministic  scenario,  the  sequence 
of  events  is  fixed  and  unchanged  by  the  actions  of  any 
human  in  the  loop.  More  complex  scenarios  may  have 
probabilistic  rules  that  generate  variable  events  for 
given  sets  of  scenario  conditions  and  may  also  change  the 
sequence  in  response  to  actions  of  the  human. 

Models  are  used  in  human-in-the-loop  simulations  to 
represent  endogenous  and  exogenous  variables  and  processes 
of  the  system  being  simulated.  For  example,  atmospheric 
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models  may  be  used  to  generate  weather  conditions  or 
vehicular  models  may  be  used  to  generate  movement  charac¬ 
teristics  of  an  aircraft  or  ground  vehicle. 

In  addition  to  a  functional  representation  of  a  system 
and  environmental  effects,  a  simulator  also  includes 
subsystems  for  the  control  of  the  simulator,  means  for 
monitoring  the  progress  of  an  exercise,  instructional 
support  features  in  the  case  of  training  simulators,  and 
performance  measurement  capabilities.  The  control  station 
for  the  early  Link  flight  trainers  consisted  of  a  desk 
with  a  few  instrument  displays  and  a  ground  path  plotter 
(Faconti,  1979).  Since  that  time,  control  and  instructor 
stations  have  evolved  into  rather  complex  devices  in  their 
own  right,  with  numerous  capabilities  for  controlling  and 
directing  simulator  exercises.  Soon  after  digital  com¬ 
puters  became  common  in  simulators,  discrete  indicators 
and  controls  for  various  functions  proliferated.  More 
recently  the  tendency  has  been  to  reduce  the  number  of 
displays  by  using  cathode  ray  tubes  that  are  capable  of 
displaying  a  variety  of  images,  and  to  use  conventional 
keyboards  and  multifunction  switches  for  simulator 
control. 

Training  simulators,  particularly  the  more  complex 
devices,  include  a  number  of  features  for  training 
purposes,  such  as  automated  demonstrations,  automated 
cueing,  manual  and  programmable  sets  of  initializing 
conditions  and  malfunctions,  freeze  {the  ability  at  any 
point  to  stop  the  action  and  later  continue  the  simula¬ 
tion),  recording  and  replay  of  the  simulation  exercise, 
and  hard  copy  printouts  of  performance  information  (Semple 
et  al.,  1981a).  Engineering  simulators  generally  have 
similar  control  capabilities  plus  additional  features  to 
permit  changes  in  the  basic  algorithms  driving  the 
simulation. 


INVESTMENT  IN  RESEARCH  AND  DEVELOPMENT  OF  SIMULATORS 

The  relative  dearth  of  behavioral  research  relevant  to 
simulator  design  and  use  is  apparent  from  a  review  of  the 
recent  research  funding  patterns  of  the  Department  of 
Defense,  the  government  agency  that  provides  the  largest 
amount  of  research  funding  and  is  also  the  largest  user 
of  simulators. 

In  the  report  of  the  Joint  Directors  of  Laboratories 
Technology  Initiative  Panel  for  Training  and  Simulation, 
Alluisi  (Alluisi  et  al.,  1983)  noted  that  the  total  DoD 
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investment  in  research  and  technology  development,  which 
includes  the  Basic  Research  Exploratory  Development  and 
Advanced  Development  budget  categories,  was  $7.06  billion. 
Funds  for  training  and  simulation  research  and  development 
performed  by  government  laboratories,  including  awards  to 
outside  contractors,  totaled  $62.7  million,  or  0.9  percent 
of  the  DoD  technology  investment.  DoD  support  of  academic 
institutions  for  training  and  simulation  for  fiscal  1981 
and  1982  together  amounted  to  slightly  more  than  $20 
million. 

The  same  report  states  that  of  the  $62.7  million  total, 
$1.4  million,  $23.3  million,  and  $38.0  million  were  spent 
for  basic  research,  exploratory  development,  and  advanced 
development,  respectively.  During  this  same  period,  $62.3 
million  was  invested  by  industrial  firms  in  training  and 
simulation  research  and  development. 

The  Manpower  and  Training  Research  Information  System 
(MATRIS)  of  the  Defense  Technical  Information  Center  was 
queried  to  determine  the  number  and  nature  of  DoD  sup¬ 
ported  work  elements  (research  projects)  started, 
continuing,  or  ending  in  the  period  from  fiscal  1981  to 
mid-1983  in  the  simulation  and  training  devices  category, 
or  work  elements  closely  related  to  simulation  and  train¬ 
ing  devices  in  other  categories.  The  work  elements  were 
segregated  by  the  three  budget  categories  for  technology 
base  development,  i.e.,  basic  research,  exploratory 
development,  and  advanced  development.  The  number  of 
work  elements  found  by  the  MATRIS  query  are  shown  in 
Table  1. 

The  summary  descriptions  provided  by  MATRIS  of  each  of 
the  101  identified  work  elements  were  reviewed  to  discover 
which  ones  involved  at  least  a  component  of  behavioral 
research  or  formal  behavior  evaluations.  Hie  purpose  of 
this  review  was  to  make  an  approximate  determination  of 
the  number  of  DoD  work  elements  that  directly  or 
indirectly  are  likely  to  provide  behavioral  data  relevant 
to  the  design  and  use  of  simulators.  The  results  of  the 
review  are  summarized  in  Table  2.  From  this  table  it  is 
apparent  that  very  few  work  elements  (only  eight  were 
identified)  are  devoted  to  basic  research  on  simulation 
and  training  devices.  Six  of  the  eight  involve  behav- 
iorally  related  research  or  evaluation.  While  the  number 
of  work  elements  in  either  exploratory  development  or 
advanced  development  is  much  larger,  only  one-third  of 
these  work  elements  involve  a  component  of  behavioral 
research  or  formal  behavioral  evaluation.  The  vast 
majority  of  work  elements  are  equipment  or  software 
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TABLE  1  Research  on  Simulation  and  Training,  October 
1981  to  mid-1983 


Budget 

Category 

Simulation 
and  Training 
Devices 

Nonsimulation 
Training  and 
Devices 

Total 

Basic  research 

5 

3 

8 

Exploratory 

development 

37 

15 

52 

Advanced 

development 

41 

0 

41 

Total 

83 

18 

101 

NOTE:  The  table  shows  the  number  of  DoD  work  elements 
within  the  categories  of  simulation  and  training  devices 
and  closely  related  work  elements  from  the  other 
categories  that  started,  continued,  or  ended  in  the 
period  from  fiscal  1981  to  mid-1983.  Work  elements  are 
segregated  by  technology  base  development  budget  category. 


developments,  analysis  of  requirements  for  simulators  or 
training  devices,  or  demonstrations  of  technology  or 
equipment  capabilities. 

It  is  apparent  from  the  foregoing  that  the  amount  of 
funding  for  technology  base  research  and  development 
supported  by  the  Department  of  Defense  is  small  relative 
to  its  overall  research  and  development  budget.  Whether 
or  not  the  level  of  funding  for  simulation  is  adequate, 
the  number  of  research  projects  devoted  to  behavioral 
aspects  of  simulation  and  training  devices,  particularly 
basic  research,  is  very  small  compared  with  those  devoted 
to  equipment  development  and  cannot  be  considered  anything 
but  minuscule  relative  to  the  large  investments  in  and 
number  of  simulators  used  by  the  Department  of  Defense. 

At  least  three  recent  high-level  reports  (U.S.  Depart¬ 
ment  of  Defense,  Defense  Science  Board,  1982;  Herman, 

1982;  Alluisi  et  al.,  1983)  address  in  whole  or  in  part 
simulation  and  training  device  developments  and  acknowl¬ 
edge  their  importance  to  the  Department  of  Defense. 


.k;  <*• 
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TABLE  2  Research  on  Simulation  and  Training  Devices  With 
and  Without  a  Behavioral  Component 


Budget 

Category 

Behavioral 

Component 

No  Behavioral 
Component 

Total 

Basic  research 

6 

2 

8 

Exploratory 

development 

13 

39 

52 

Advanced 

development 

10 

31 

41 

Total 

29 

72 

101 

NOTE:  The  table  shows  number  of  DoD  work  elements  within 
the  categories  of  simulation  and  training  devices  involving 
or  not  involving  a  component  of  behavioral  research  or 
formal  behavioral  evaluation  by  research-level  budget 
category. 


However,  none  of  these  reports  distinguishes  between 
development  of  equipment,  hardware,  and  software  and 
development  of  behavioral  principles  for  the  design  and 
use  of  simulators.  Opportunities  and  needs  for  improve¬ 
ments  in  simulation  and  training  are  discussed  solely 
under  the  aggregate  term  "training  technology." 

Simulation  is  an  immensely  important  tool  for  the 
nation.  Billions  of  dollars  have  been  sp>ent  on  simula¬ 
tion  and  billions  more  will  be  spent  in  the  near  future 
(Deegan,  1981)  .  Tens  of  thousands  of  people  are  involved 
in  the  construction,  operation,  and  maintenance  of  simu¬ 
lators.  Although  the  military  services,  NASA,  commercial 
airlines,  and  the  aerospace  industry  were  traditionally 
the  principal  users  of  simulators,  the  use  of  this  tool 
is  growing  rapidly  in  other  areas  such  as  the  electric 
power  industry  and  in  the  development  of  complex  hardware 
and  software  systems,  particularly  those  with  significant 
human  operator  involvement  such  as  automobile  driver 
training. 

This  national  investment  continues  to  grow  because  of 
the  cost-effectiveness  and  utility  of  simulators.  For 
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example,  for  the  new  Boeing  767,  aircraft  training  time 
costs  $7 , 000-8 , 000/hour ,  as  opposed  to  average  expendi¬ 
tures  of  $400/hour  for  full  flight  simulators.  In 
addition,  the  simulator  has  provisions  for  training  in 
emergency  procedures  that  cannot  be  accomplished  safely 
in  the  aircraft  (Aviation  Week  and  Space  Technology, 
1983).  During  1980  and  1981,  the  median  flight 
simulator/aircraft  operating  cost  ratio  of  42  military 
aircraft  was  0.08,  down  from  0.12  three  years  earlier. 
Under  the  Federal  Aviation  Administration's  (FAA) 

Advanced  Simulator  Program,  airline  pilots  may  be 
certified  entirely  in  an  approved  simulator  that  meets 
specific  requirements.  A  simulator  was  approved  for 
certification  of  the  United  Airlines  B-727  and  for  other 
airlines  in  1982;  approval  of  other  airlines'  simulators 
may  soon  follow.  The  design  of  new  military  aircraft  is 
supported  by  mission  simulators  that  permit  representa¬ 
tive  military  missions  to  be  "flown"  and  thereby  reducing 
the  redesign  requirements  after  the  aircraft  is  built  and 
resulting  in  more  effective  fighting  machines  (Aviation 
Week  and  Space  Technology,  1983) . 

DEFINITION  OF  SIMULATORS 

In  general,  simulators  may  be  defined  operationally  as 
encompassing  the  following  characteristics:  (1)  A 
specific  system  or  equipment  is  represented  through 
either  artificial  duplication  of  equipment  or  embedding 
artificial  input  and  measurement  features  in  actual 
equipment.  In  the  case  of  war  games,  scenarios  alone  or 
scenarios  plus  equipment  may  be  used.  Simulators  may  be 
employed  in  contrast  to  the  use  of  the  actual  equipment. 
(2)  Simulators  not  only  duplicate  equipment,  but  they  are 
also  used  to  create  and  control  external  events  such  as 
malfunctions,  targets,  and  environmental  effects.  (3) 

The  system  is  interactive  in  that  human  input  results  in 
changes  to  the  equipment  and  equipment  changes  are 
responded  to  by  the  operator. 


ORIGINS  OF  SIMULATION  AND  SIMULATION  RESEARCH 

The  precursors  to  simulators  were  the  primitive 
devices  used  as  flight  trainers  before  World  War  I.  The 
mechanical  Link  trainer  developed  before  World  War  II  for 
instrument  flight  training  was  a  generic  device,  i.e.,  it 
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did  not  represent  any  specific  aircraft.  A  simulator  of 
the  Lockheed  Hudson  aircraft,  developed  in  1942,  was  the 
first  simulation  of  a  specific  aircraft  (Whiteside,  1983). 
Modern  electronic  simulators  were  first  used  in  the  early 
1950s  by  commercial  airlines  for  the  PanAm  B-377  Strato- 
cruiser  (Brice,  1951)  .  This  airline  simulator  was  fol¬ 
lowed  closely  by  devices  for  the  U.S.  Air  Force's  B-50D 
strategic  bomber  (Dice,  1953)  and  the  U.S.  Navy's  SNJ 
trainer  (Wilcoxon  et  al.,  1954).  These  early  simulators 
had  fixed  (nonmoving)  bases.  Sophisticated  visual 
systems  and  complex  motion  bases  appeared  soon  there¬ 
after.  There  was  a  proliferation  of  training  simulators 
in  other  areas,  including  ground  transportation,  mining, 
maritime  activities,  space,  command  and  control  functions, 
and  training  of  control  room  operators  in  nuclear  power 
plants. 

As  early  as  the  mid-1950s,  there  were  several  signifi¬ 
cant  reports  on  the  human  factors  aspects  of  simulators 
that  continue  to  be  classics  in  the  field.  Six  of  these 
are  cited  here  with  brief  annotations  of  their  particular 
value . 

(1)  Robert  Gagne,  "Training  Devices  and  Simulators: 
Some  Research  Issues."  American  Psychologist,  August 
1954.  An  early  and  insightful  discussion  of  training 
research  issues. 

(2)  Robert  B.  Miller,  Psychological  Considerations  in 
the  Design  of  Training  Equipment,  WADC  Technical  Report 
54-563.  Wright  Air  Development  Center,  Ohio,  December 
1954.  Identified  the  difference  between  psychological 
and  engineering  fidelity  and  the  implications  for  design. 

(3)  Flight  Simulator  Utilization  Handbook,  HFORL 
Report  No.  43.  Human  Factors  Operations  Research 
Laboratories,  Bolling  Air  Force  Base,  Washington,  D.C., 
August  1953.  Recognized  that  the  effectiveness  of  a 
simulator  is  dependent  on  utilization  techniques  and 
showed  how  quality  training  programs  could  be  developed. 

(4)  H.C.  Wilcoxon,  E.  Davy,  and  J.C.  Webster, 
Evaluation  of  the  SNJ  Operational  Flight  Trainer, 

Technical  Report  SPECDEVCEN  999-2-1.  Port  Washington, 
N.Y.,  22  March  1954.  Transfer  of  training  experiment 
with  early  Navy  electronic  simulator. 

(5)  H.M.  Parsons,  Man-Machine  System  Experiments. 
Baltimore,  Md.:  Johns  Hopkins  Press,  1972. 

Documentation  of  training  and  research  use  of  simulators 
primarily  for  command,  control,  and  communication  systems. 
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(6)  Adorian  et  al.,  50  Years  of  Flight  Simulation. 
Royal  Aeronautical  Society  Network,  Buckingham,  England, 
1979.  An  outline  of  the  evolution  of  flight  simulators 
from  their  early  inception  to  modern  devices. 

Other  important  reports  are  U.S.  Air  Force,  1978;  NATO 
Advisory  Group  for  Aerospace  Research  and  Development, 
1980b;  U.S.  Air  Force,  1982;  and  Richards  and  Dismukes, 

1982. 
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THE  USES  OF  SIMULATION 


The  primary  use  of  simulators  is  for  training.  How¬ 
ever,  they  are  being  used  increasingly  for  design, 
development,  and  evaluation  of  systems;  analysis  and 
evaluation  of  standards  and  procedures;  and  conduct  of 
basic  and  applied  research.  The  large  number  and  variety 
of  simulators  in  use  in  the  United  States  make  it 
virtually  impossible  to  compile  a  comprehensive  list. 
Furthermore,  simulators  are  rarely  treated  as  a  separate 
category  for  purposes  of  inventory,  but  are  treated  as 
integral  parts  of  the  facilities  in  which  they  are  used 
and  classified  according  to  their  use.  Training  simu¬ 
lators,  for  example,  are  normally  classified  under  the 
more  general  category  of  training  devices.  Engineering 
simulators  are  used  for  system  development  and  research 
simulators  are  regarded  as  laboratory  equipment. 

Tables  3  and  4  illustrate  the  proliferation  of  simu¬ 
lator  types  and  uses,  and  Appendix  A  describes  particular 
simulators  in  five  categories  of  simulation,  which  differ 
either  in  their  principal  purpose  or  major  physical 
characteristics:  (1)  design,  development,  and  evaluation 

simulators — in  short,  engineering  simulators;  (2)  training 
simulators;  (3)  research  simulators;  (4)  battle  simu¬ 
lators;  and  (5)  embedded  simulation. 

The  first  two  columns  of  Table  3,  labeled  "Simulator 
Types,"  show  the  kind  of  equipment  and  activities  that 
are  simulated  in  the  military  and  civil  sectors.  The 
third  column,  labeled  "Cognizant  Government  Agency," 
indicates  the  branch  of  government  that  has  cognizance 
over  or  support  of  the  activities  in  a  sector.  Table  4 
lists  the  principal  uses  of  simulators. 

A  cross-indexing  of  each  type  and  use  has  not  been 
made  because  we  did  not  conduct  an  inventory;  an 
exhaustive  listing  of  every  particular  simulator  within 
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TABLE  3  Simulator  Types 


Types  of  Sumulators 

Cognisant 

General 

Specific 

Government 

Agency 

Aircraft 

General  aviation 

FAA 

Commercial 

FAA 

Military 

DoD 

Spacecraft 

DOD/NASA 

Surface  transport 

Automobiles 

DoT/DoD 

Trucks 

DoT/DoD 

Locomotives/trains 

DoT 

Civil  marine 

Propulsion  systems 

DoD 

transport 

Collision  avoidance 

DoT 

Marine  warfare 

Subsurface 

DoD 

Surface 

DOD 

Command  and  control 

Military 

DOD 

Space 

DOD/NASA 

Civil  air  traffic 
Industrial 

FAA 

Tactical  surface 

Direct  fire 

DoD 

Warfare 

Tanks 

DOD 

Artillery 

DOD 

Missiles 

DoD 

Electronic 

DoD 

Battle 

DoD 

Tactical/strategic 

Ranges 

DoD 

Air  warfare 

Air  combat  simulators 

DoD 

Energy  systems 

Nuclear 

DoE/NRC 

Mining 

BuMINES 

Environmental 

Centrifuge 

DoD 

Ejection 

DoD 

Diving 

DoD 

Motion  base 

DoD 

Fire  bightlng 

DoD 

Medical 

DoD 

FAA: 

DoD; 

NASA! 

DoT: 

DoE  i 
NRCt 

BuMlnes; 


Federal  Aviation  Administration 
Department  of  Defense 

National  Aeronautics  and  Space  Administration 

Department  of  Transportation 

Department  of  Energy 

Nuclear  Regulatory  Commission 

Bureau  of  Mines 
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TABLE  4  Typical  Uses  of  Simulators 


Research 
Workload 
Decision-making 
Performance  assessment 
Stressor  effects 
Visual/motion  systems 
Criterion  development 

Design  and  Development 
Concept/system  demonstration 
Parametric  studies 
Alternate  configurations 
Subsystem  evaluation 
System  evaluation 
Aircrew  advisory 
Prototype  assessments 
Procedures  development 
Tactics  development 
Mission  capability  assessment 
Modification  studies 

Test  and  Evaluation 
Subsystem/system 
Operational  capability 
Fly-offs 

Operations  Crew 
Initial  training 
Instrument  training 
Normal  procedures  training 
Emergency  procedures  training 
Transition  training 
Refresher  training 
Tactics  training 
Mission/battle  training 
Individual/team  assessment 
Accident  investigation 
Selection-initial/operations 

Maintainer 
Procedures  training 
Troubleshooting  training 
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type,  if  feasible,  would  fill  a  large  book.  The  purpose 
of  the  table  is  to  illustrate  the  pervasiveness  of 
simulation. 


TRAINING 

The  forerunners  of  modern,  interactive  simulators  were 
the  flight  trainers  developed  in  the  1920s  and  1930s 
(Adorian  et  al.,  1979).  The  Link  Trainer,  used  by  the 
Army  Air  Corps  for  instrument  flight  training,  and  its 
successors  were  viewed  as  devices  that  could  substitute 
on  a  restricted  basis  in  the  training  of  a  limited  number 
of  tasks  and  skills  for  actual  equipment.  Early  trainers 
were  generalized  representations  of  aircraft;  their 
dynamics  were  like  those  of  an  aircraft,  but  no  particular 
one,  and  the  kind  of  training  that  could  be  accomplished 
was  somewhat  restricted.  What  could  not  be  learned  in  a 
device  or  learned  to  the  level  of  proficiency  required 
was  learned  during  practice  with  operational  equipment. 

In  the  late  1940s  flight  trainers  evolved  from  generalized 
devices  to  simulators  of  specific  aircraft.  Because 
their  appearance  and  their  flight  dynamics,  implemented 
by  analog  computers,  were  faithful  representations,  more 
tasks  could  be  trained  to  the  levels  of  proficiency 
required  operationally. 


Engineering  Advances 

From  the  1950s  through  the  early  1970s,  the  prominent 
concerns  were  mostly  engineering  issues  related  to 
extending  the  range  of  tasks  that  could  be  performed  or 
trained  in  simulators  and  to  providing  greater  fidelity 
and  realism  in  the  simulation  of  system  functions  and 
environmental  effects.  Engineering  advances  in  simulation 
have  been  closely  linked  to  the  evolution  of  mechanisms 
to  produce  the  operational  functions,  especially  in 
aircraft.  Mechanical  and  electromechanical  methods  were 
supplanted  by  analog  computers  and,  finally,  by  digital 
computers  that  are  the  heart  of  all  modern  simulators. 

It  is  interesting  to  note  that  the  minicomputers  in 
common  use  today  had  their  origins  in  the  Universal 
Digital  Operational  Flight  Trainer  project,  which 
developed  a  digital  computer  to  solve  aircraft  dynamic 
equations.  Progress  in  computer  technology,  especially 
in  processing  speed  and  memory  capacity,  permitted 
simulation  of  multiple,  complex  functions  in  real  time. 
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Interrelated  developments  In  display  technology 
allowed  the  development  of  visual  displays  portraying 
detailed  scenes  of  the  outside  world  as  well  as  simulation 
of  sensor  displays.  Model  terrain  boards  and  television 
systems,  which  have  been  used  extensively  in  aircraft, 
automobile,  and  ship  simulators  to  create  scenes  of  the 
external  environment,  are  rapidly  being  replaced  by 
computer-generated  imagery  systems.  The  same  systems  are 
being  used  for  simulating  displays  of  infrared  imagery  as 
well  as  radar  and  sonar  returns.  Advances  in  displays 
also  permitted  the  portrayal  of  active  elements,  e.g., 
other  aircraft  or  ground  vehicles. 


Part-Task  Simulators 

Many  modern  simulators  can  support  a  wide  range  of 
tasks;  however,  because  of  their  complexity,  these  devices 
are  also  quite  expensive.  For  some  training  purposes  it 
is  not  cost-effective  to  use  complex  devices  for  training 
a  limited  set  of  tasks  or  simple  tasks,  such  as  pro¬ 
cedures;  the  full  range  of  capabilities  of  a  simulator 
may  not  be  necessary.  And  when  a  large  number  of  people 
needs  to  be  trained,  multiple  large  simulators  may  not  be 
affordable.  For  these  reasons,  less  expensive  part-task 
simulators  have  been  developed. 

The  availability  of  minicomputers  and  microcomputers 
has  greatly  facilitated,  the  development  of  these  devices. 
Part-task  trainers  can  be  designed  to  perform  only  those 
functions  necessary  for  a  specific  purpose.  For  example. 
Navy  Device  2C62  is  a  low  cockpit  procedures  trainer  for 
the  SH-3H  helicopter  (Caro  et  al.,  1984).  This  device 
has  only  a  few  instruments  and  dials  that  operate,  and 
the  dial  needles  move  in  discrete  steps  rather  than  smooth 
movements;  other  indicators  are  only  photographs  of 
instrument  faces.  Cockpit  dimensions  are  approximate 
rather  than  exact.  Cues  indicate  onset  of  malfunction 
but  not  progressive  degradation.  Also  standard  chairs 
rather  than  cockpit  seats  are  used.  The  control  station 
for  the  simulator  is  a  conventional  computer  terminal 
consisting  of  a  cathode  ray  tube  display  and  keyboard. 

Nonrealistic  or  schematic  representations  of  system 
functions  that  still  allow  operator  interaction  are 
another  form  of  low  cost,  part-task  training  simulators. 
The  simulation  may  occur  entirely  on  a  computer  terminal. 
The  propulsion  plant  simulator  STEAMER,  described  in 
Appendix  A,  is  an  example  of  this  type  of  device.  The 
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important  aspect  of  this  type  of  simulator,  other  than 
its  low  cost,  is  that  it  represents  the  function  of  a 
system  in  a  way  that  is  more  easily  understood  than  would 
he  the  case  it  the  simulator  were  a  physical  representa¬ 
tion  of  the  actual  power  plant.  This  example  vividly 
illustrates  the  fact  that  knowledge  of  behavioral  prin¬ 
ciples,  i.e.,  how  people  acquire  information  and  develop 
mental  representations,  greatly  extends  the  range  of 
possibilities  for  simulator  design  beyond  achieving 
physical  realism. 


ENGINEERING  DESIGN  AND  RESEARCH 

Although  training  has  been  the  principal  application 
of  simulators,  they  have  also  been  used  for  equipment 
design,  development  of  operating  procedures,  and  research 
on  human-machine  interaction  since  at  least  World  War 
II.  The  first  simulation  laboratory  for  such  purposes 
appears  to  have  been  the  one  established  at  Beavertail 
Point,  Rhode  Island,  in  1945  under  the  sponsorship  of  the 
National  Defense  Research  Committee  (Parsons,  1972). 
Simulated  shipboard  combat  information  centers  were  used 
to  test  new  equipment,  methods,  and  procedures  for 
plotting  hostile  aircraft  paths  and  communicating  this 
information  to  antiaircraft  directors.  In  the  late 
1940s,  a  1-CA-l  Link  flight  training  device  at  the 
University  of  Illinois,  Aviation  Psychology  Laboratory, 
was  used  for  the  development  of  pictorial  displays  for 
aircraft  (Williams  and  Roscoe,  1950).  In  1950  an  air 
traffic  control  simulator  was  built  at  the  Technical 
Development  Center  of  the  Civil  Aeronautics  Administra¬ 
tion  in  Indianapolis  to  study  terminal  area  approach 
procedures,  display  design,  and  human  factors  issues. 
Parsons  (1972)  describes  over  30  large-scale  simulation 
studies  of  human-machine  interaction  that  occurred 
between  1948  and  1966  in  which  simulators  were  used  for 
projects  ranging  from  investigations  of  team  composition 
to  equipment  design  for  Apollo  lunar  missions. 

Today  simulators  for  engineering  design  and  research 
are  common  in  government  laboratories,  universities,  and 
civilian  industries.  Engineering  applications  are  as 
diverse  as  the  development  of  command-and-control  equip¬ 
ment  and  the  assessment  of  ride  quality  of  farm  machinery. 
One  indication  of  their  importance  and  widespread  use  is 
the  fact  that  Aviation  Week  and  Space  Technology,  a  major 
aerospace  industry  trade  publication,  featured  in  its 
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January  17,  1983,  edition  12  articles  on  the  use  of  simu¬ 
lators  in  aircraft  design. 


LICENSING  AND  CERTIFICATION 

A  relatively  recent  but  important  use  of  simulators  is 
for  licensing  and  certification.  In  1980  the  Federal 
Aviation  Administration  issued  a  rule  titled  Advanced 
Simulation  (Federal  Register,  June  30,  1980),  which 
permits  all  operational  training  and  crew  member  cer¬ 
tification  (except  preflight  checks)  to  be  conducted  in  a 
simulator  that  meets  certain  requirements  and  is  part  of 
an  approved  training  program.  In  effect,  a  simulator 
meeting  the  requirements  is  a  substitute  for  the  use  of 
an  aircraft  to  qualify  personnel.  In  1982  the  first 
simulator  to  receive  FAA  approval  for  training  and  crew 
certification  was  a  Boeing-727  simulator  operated  by 
United  Airlines.  This  milestone  vividly  documents  the 
high  degree  of  fidelity  and  realism  that  can  be  achieved 
in  simulators.  It  also  indicates  the  degree  to  which 
simulators  have  been  accepted  and  are  relied  upon  for 
training  purposes  (see  also  "Airplane  Simulator  and 
Visual  System  Evaluation,"  Advisory  Circular  AC  No. 

120-40,  Federal  Aviation  Administration,  Washington, 

D.C.,  January  31,  1983). 

Only  aircraft  simulators  are  used  currently  for 
certification.  However,  the  Nuclear  Regulatory  Commission 
is  developing  requirements  to  certify  nuclear  power  plant 
control  room  simulators  for  operator  training  (Rankin  et 
al.,  1984).  Ultimately  these  simulators  may  have  the 
capability  to  perform  certification  tests  for  nuclear 
power  plant  operators. 

NUMBERS  AND  COSTS  OF  SIMULATORS 

The  military  services  are  the  largest  users  of  simu¬ 
lators  with  the  greatest  financial  investment.  They  will 
request  a  total  of  over  $6  billion  for  simulators  and 
training  devices  between  1982  and  1986,  and  the  rate  of 
expenditure  will  increase  annually  (Deegan,  1981) . 
Approximately  35  percent  of  these  expenditures  will  be 
for  flight  simulators.  NASA  has  approximately  30  large 
simulators,  mostly  aircraft  simulators,  with  an  estimated 
total  value  of  $100  million. 
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The  Army's  Index  of  Training  Devices  (U.S.  Department 
of  the  Army,  1980)  contains  a  list  of  approximately  390 
types  of  training  devices.  Over  300  of  these  are  simu¬ 
lators  and,  since  there  are  often  multiple  copies  of 
devices,  the  actual  number  of  individual  devices,  each 
costing  from  a  few  thousand  to  several  million  dollars, 
is  probably  in  the  thousands.  The  Army  has  spend  nearly 
$140  million  thus  far  on  the  development  and  procurement 
of  its  Multiple  Integrated  Laser  Equipment  System  (MILES) 
(Remhotz,  1982).  This  system  simulates  and  'ecords  the 
effects  of  direct  engagements  between  small  arms,  anti¬ 
tank,  and  tank  weapons;  it  is  used  to  train  infantry  and 
combined  arms  units  up  to  battalion  size.  The  eventual 
procurement  cost  to  equip  each  of  the  16  active  Army 
divisions  and  the  National  Training  Center  is  projected 
to  approach  $1  billion. 

The  Navy's  Index  of  Training  Devices  (U.S.  Department 
of  the  Navy,  1981)  contains  approximately  2,700  individual 
training  devices,  primarily  simulators.  Privately  com¬ 
piled  lists  of  military  aircraft  simulators  (Defense 
Market  Survey,  1980a,  1980b)  show  that  the  Navy  and  the 
Air  Force  have  462  and  232  of  these  multimillion-dollar 
devices,  respectively.  A  recent  two-cockpit  Naval  air 
combat  simulator.  Device  2E6,  costs  over  $25  million. 
Training  simulators  for  the  U.S.  Marine  Corps  AV-8B 
aircraft  consist  of  three  devices;  a  part-task  procedures 
trainer,  an  operational  flight  trainer,  and  a  weapons 
system  trainer;  one  set  costs  nearly  $50  million.  It 
should  be  noted  that  research  and  development  costs 
frequently  inflate  the  cost  of  a  simulator  when  procured 
as  a  prototype  or  on  a  one-of-a-kind  basis.  Currently, 
B-52  weapons  systems  trainer  simulators  cost  about  $44 
million  each.  The  B-l  simulator  is  estimated  to  cost  $75 
million  (Burpee,  1981)  . 

Worldwide  there  are  about  290  large  commercial  aircraft 
simulators  operated  principally  by  airlines,  each  device 
costing  up  to  $10  million  (Whitaker,  1981).  There  are  19 
existing  nuclear  power  plant  simulators  costing  approxi¬ 
mately  $7  million  each,  and  16  more  are  on  order  (Rankin 
et  al. ,  1984) . 

Procurement  of  maintenance  trainers,  both  simulators 
and  actual  equipment,  by  the  Department  of  Defense  has 
increased  since  1977.  Current  annual  procurement  costs 
are  estimated  to  be  about  $120  million.  Expenditures  for 
maintenance  simulators  and  actual  maintenance  equipment 
trainers  in  the  military  services  were  projected  to  be 
$620  million  between  1975  and  1985  (U.S.  Department  of 
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Defense,  1982) .  About  two-thirds  of  this  procurement  is 
for  maintenance  of  aircraft;  the  remainder  is  divided 
among  all  other  areas  of  maintenance  of  military 
equipment. 

Another  large  use  of  simulators  is  for  development  and 
evaluation  of  command,  contro) .  and  communication  (C3) 
systems.  Currently  the  Depai.  ient  of  Defense  has  41 
operational  C3  manned  test  beds  with  another  16  either 
under  procurement  or  proposed  (Gasparotti  et  al.,  1981). 

The  number  of  personnel  required  to  support  simulators 
is  also  substantial.  The  Navy  estimates  it  has  2,200 
people  employed  in  the  operation  and  maintenance  of  300 
major  simulation  devices.  The  McDonnell  Douglas  Aircraft 
Company  employs  approximately  150  support  personnel  for 
its  Flight  Simulation  Facility,  which  consists  of  five 
separate  cockpit  stations  sharing  common  computer  and 
imagery  sources  and  is  used  for  engineering  design  and 
development  of  fighter  aircraft. 


NEEDED  TECHNICAL  IMPROVEMENTS 

Although  simulators  have  reached  an  advanced  state  of 
technical  maturity,  there  are  still  some  areas  where 
technical  improvements  are  desired.  The  succeeding 
chapters  of  this  report  largely  focus  on  the  importance 
of  behavioral  science  and  human  factors  contributions  to 
simulator  design  and  use;  the  remainder  of  this  chapter 
provides  a  brief  discussion  of  technological  and  engineer¬ 
ing  advancements  needed  to  extend  the  capabilities  of 
simulators. 


Visual  Simulation 

One  of  the  most  prominent  technical  concerns  is 
improvement  of  the  quality  of  visual  scenes  for  military 
flight  training  simulators.  For  certain  applications, 
greater  detail  and  realism  of  visual  scenes  is  thought  to 
be  necessary  to  support  such  tasks  as  low-level,  high¬ 
speed  flight  for  military  aircraft  (Richards  and  Dismukes, 
1982)  and  nap-of-the-earth  flight  by  helicopters.  The 
judgments  of  experienced  pilots  as  well  as  performance 
tests  have  shown  that  current  visual  system  technology  is 
not  sufficient  to  provide  the  distance  and  altitude  cues 
required  for  these  flight  regimes.  While  one  possibility 
is  to  make  better  use  of  the  capabilities  of  existing 
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visual  systems,  it  is  commonly  believed  that  simulated 
low-level  flight  will  be  possible  only  by  providing 
greater  detail,  such  as  ground  vegetation  and  texture. 


Voice  Communications 

Technical  advances  are  desired  in  voice  communica¬ 
tions,  both  the  production  and  recognition  of  human 
speech.  In  this  area  the  problems  are  producing  variable 
content  messages  as  if  spoken  by  a  human  and  recognizing 
what  is  said  by  participants  so  that  appropriate  simulated 
events  or  actions  can  occur.  For  example,  in  training 
air  traffic  controllers,  it  would  be  desirable  to  have 
simulated  pilots  who  can  engage  in  two-way  communication 
with  the  controller  and  respond  appropriately  to  the 
controller's  directives.  Similar  needs  exist  in  command- 
and-control  and  battle  simulation  devices,  in  which 
receiving  and  producing  voice  messages  are  primary 
activities. 


System  Models 

A  third  area  of  concern  is  improving  the  comprehensive¬ 
ness,  accuracy,  and  resolution  of  system  function  models 
for  both  normal  and  abnormal  modes  of  operation.  Almost 
all  simulator  functions  are  an  implementation  of  a  simpli¬ 
fied  or  superficial  model  of  actual  system  functions. 

For  example,  models  used  in  nuclear  power  plant  simulators 
do  not  allow  for  all  possible  interactions  between  system 
variables.  A  simple  model  of  normal  operation  of  the 
plant  may  be  adequate  for  training  operators  in  standard 
procedures.  However,  in  training  emergency  procedures,  a 
simplified  model  of  the  power  plant  operation  may  unduly 
restrict  the  kinds  of  failures  that  can  be  simulated,  or 
worse,  produce  effects  that  are  not  equivalent  to  what 
would  actually  occur  in  a  real  power  plant.  The  general 
problem  is  that  real-time  simulation  always  involves  a 
trade-off  between  the  amount  of  computation  that  can  occur 
in  a  given  time  frame,  the  complexity  of  the  functions  to 
be  computed,  and  the  cost  of  doing  so  (Belsterling  et 
al.,  1982).  Another  approach  is  to  partition  the  software 
architecture  into  submodels  that  emulate  smaller  segments 
of  the  system  in  greater  detail. 

When  system  models  are  implemented  on  digital  com¬ 
puters,  the  model  of  functions  that  are  continuous  in  the 
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real  world  may  be  updated  at  discrete  intervals.  The 
update  interval  of  a  function  can  affect  performance 
whether  or  not  these  differences  are  perceived.  For 
example,  the  equations  simulating  performance  of  the 
engines  of  the  T-2C  aircraft  in  the  Naval  Training 
Equipment  Center's  Visual  Technology  Research  Simulator 
are  updated  7.5  times  per  second.  Pilots  frequently 
comment  that  the  response  of  the  simulated  engines  to 
throttle  changes  seems  different  than  in  the  actual 
aircraft.  An  experiment  in  which  the  engine  equations 
were  updated  30  times  a  second  was  found  to  be  sub¬ 
jectively  more  realistic  to  the  pilots  and  also  yielded  a 
small  but  reliable  improvement  in  performance  as  well 
(Westra  et  al.,  1981). 


Communication  Links  Among  Simulators 

Simulators  have  been  developed  that  involve  multiple 
scenario  components  such  as  aircraft  crews,  ships,  bridge 
teams,  and  command  center  staffs.  It  is  now  considered 
desirable  to  extend  simulation  to  involve  multiple  com¬ 
ponents  at  dispersed  sites  that  act  either  cooperatively 
or  as  adversaries.  For  example,  it  would  be  desirable  to 
have  attack  aircraft  simulators  linked  through  telecom¬ 
munications  with  tactical  air  control  simulators  to 
conduct  cooperative  exercises.  Another  alternative  is  to 
link  attack  aircraft  simulators  with  air  defense  weapon 
simulators  to  permit  both  sides  to  practice  against  a 
realistic  adversary.  Attention  is  also  being  given  to  a 
network  that  would  permit  as  many  as  1,000  players,  e.g. , 
tanks,  antitank  weapons,  and  aircraft  and  antiaircraft 
weapons,  to  engage  each  other  in  realistic  but  simulated 
attacks  on  a  100-mile-by-100-mile  terrain  area. 


Low-Cost  Simulation 

Development  of  low-cost  simulators  that  can  be 
produced  in  large  numbers  is  of  particular  concern  to  the 
Department  of  Defense  (U.S.  Department  of  Defense,  Defense 
Science  Board,  1982).  These  devices  would  take  advantage 
of  the  availability  of  inexpensive  microcomputers  and 
relatively  new  technology  developments  such  as  video 
disks  and  speech  synthesis  and  recognition  systems.  Time¬ 
sharing  of  a  single  processor  among  several  trainees/ 
operator  stations  is  another  approach.  It  is  particularly 
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applicable  to  simulation  that  is  predominantly  information 
flow  rather  than  equipment  operation.  This  approach  has 
been  used  for  some  of  the  Army's  war  games  simulators. 

Another  means  for  achieving  relatively  low-cost  simu¬ 
lation  is  by  using  embedded  simulation  in  operational 
equipment.  Since  most  modern  systems  are  computer-based 
devices,  embedded  simulation  can  be  accomplished  with 
software  and  little  additional  hardware,  an  important 
factor  when  space  is  at  a  premium  such  as  in  aircraft  or 
on  ships.  Using  operational  equipment  as  a  training 
device  allows  maintenance  of  skills  or  refresher  training 
in  the  operational  context  and  reduces  the  need  to  conduct 
training  at  special  facilities. 

The  approaches  described  above  for  improving  simulation 
technology  reflect  a  general  desire  to  extend  the  scope 
of  simulation  and  increase  its  effectiveness  while  at  the 
same  time  minimizing  costs.  Extending  the  comprehensive¬ 
ness  of  simulation  by  linking  remotely  located  simulators 
and  lowering  the  cost  of  simulation  by  the  use  of  simple 
devices  and  embedded  simulation  raises  questions  about 
what  degree  of  comprehensiveness  is  necessary,  how  much 
physical  fidelity  is  enough,  and  in  what  ways  is  it 
desirable  and  effective  to  depart  from  physical  fidelity 
to  achieve  the  intended  purpose.  Answers  to  these 
questions  must  come  from  a  consideration  of  the  behavioral 
and  not  the  physical  aspects  of  simulation. 
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BEHAVIORAL  FACTORS  IN  SIMULATON 


At  this  stage  of  advanced  simulator  technology,  human 
behavior,  rather  than  equipment,  has  become  the  central 
concern  in  simulation.  Improvements  in  the  design  and 
use  of  simulators  are  more  likely  to  depend  on  greater 
development  and  use  of  behavioral  knowledge  than  on 
advances  in  engineering  technology.  In  terms  of  behavior, 
the  primary  goals  of  simulation  are  facilitating  the 
learning  that  transfers  to  the  operational  context  in  the 
case  of  training  simulators  and  eliciting  valid  per¬ 
formance  in  the  case  of  simulators  used  for  design, 
research,  or  certification  purposes. 

In  this  chapter,  the  behavioral  bases  for  simulator 
design  and  measures  of  effectiveness  are  reviewed.  Simu¬ 
lator  validity  is  advanced  as  a  more  productive  concept 
than  fidelity  as  a  criterion  for  evaluating  simulator 
requirements  and  as  a  focus  for  research  on  simulation. 
This  chapter  is  concerned  with  the  need  to  improve  our 
understanding  of  behavioral  processes  relevant  to  simu¬ 
lation.  Few  of  the  many  recommendations  for  fundamental 
research  made  during  the  past  30  years  have  been  acted  on 
(Gagne,  1954;  Miller,  1954;  Wilcoxon  et  al.,  1954; 

Parsons,  1972)  .  A  brief  review  of  current  DoD  research 
in  the  area  of  simulation  and  training  devices  indicates 
that  relatively  little  effort  is  directed  toward  funda¬ 
mental  behavioral  issues  in  simulation.  Analysis  of 
cognitive  processes  and  development  and  use  of  human 
performance  models  are  seen  as  two  of  the  potentially 
most  effective  means  for  developing  and  expressing 
behavioral  knowledge  necessary  to  enhance  simulator 
design  and  application. 
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PHYSICAL  FIDELITY  AS  A  DESIGN  GOAL 

As  described  in  Chapter  2,  the  evolution  of  simulation 
has  been  primarily  a  matter  of  technological  advancements 
to  make  simulators  more  realistic,  accurate,  and  compre¬ 
hensive  representations  of  a  particular  item  of  equipment 
or  system.  There  are  good  reasons  why  simulators  should 
be  realistic  and  comprehensive  representations  of  real 
equipment  and  systems.  The  more  like  the  real-world 
counterpart,  the  greater  is  the  confidence  that  perfor¬ 
mance  in  the  simulator  will  be  equivalent  to  operational 
performance  and,  in  the  case  of  training,  the  greater  is 
the  assurance  that  the  simulator  will  be  capable  of 
supporting  the  learning  of  the  relevant  skills.  Everyday 
experience  confirms  that  a  simulator  of  sufficient 
fidelity  will  produce  learning  or  performance  approxi¬ 
mating  that  which  would  be  expected  if  actual  equipment 
were  used. 

From  a  practical  standpoint,  striving  for  realism  and 
comprehensiveness  greatly  simplifies  the  simulator  design 
problem.  Designing  a  simulator  to  realistically  and  com¬ 
prehensively  duplicate  a  real-world  item  of  equipment  or 
system  is  a  matter  of  achieving  physical  and  functional 
correspondence.  The  characteristics  of  the  human  par¬ 
ticipant  can  be  largely  ignored.  When  compromises  in 
realism  are  necessary  due  to  technical  limitations,  they 
are  based  on  physical  rather  than  behavioral  criteria  and 
commonsense  interpretations  of  their  consequences. 

However,  there  are  limitations  and  cost  penalties 
associated  with  relying  on  the  duplication  of  physical 
characteristics  as  a  design  guideline.  The  degree  of 
duplication  possible  has  a  natural  limit — the  operational 
system.  In  most  cases,  perfect  duplication  is  limited 
not  so  much  by  the  characteristics  of  the  target  system 
but  by  characteristics  of  the  operational  environment  or 
events.  Simulating  the  outside  environment  and  events 
that  affect  or  drive  the  simulation  can  be  difficult  and 
expensive,  and  small  gains  in  realism  often  can  be 
achieved  only  at  relatively  great  incremental  costs.  For 
example,  realistically  simulating  visual  scenes  is  still 
one  of  the  most  challenging  technical  problems,  and  the 
visual  system  is  often  the  most  costly  component  of  a 
simulator.  In  addition,  when  the  operational  environment 
includes  complex  active  elements,  e.g.,  opposing  and 
friendly  units  in  battle  simulation  or  multiple  aircraft 
in  an  air  traffic  control  situation,  simulating  the 
actions  and  communications  is  very  difficult. 
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Another  limitation  is  that  evaluation  of  features  of  a 
simulator  is  a  matter  of  relating  specific  characteristics 
to  specific  task  performance  outcomes.  Under  these  cir¬ 
cumstances  it  is  not  possible  to  generalize  about  or 
predict  what  features  will  be  useful  in  another  simulation 
context,  or  how  a  person  will  perform  if  the  task  is 
changed. 

The  premise  that  precise  physical  duplication  is 
important  has  commonsense  appeal.  However,  this  premise 
has  no  theoretical  foundation  to  predict  how  departures 
from  realism  and  comprehensiveness  might  affect  simulator 
effectiveness. 


THE  BEHAVIORAL  APPROACH  TO  SIMULATION 

As  stated  above,  the  premise  in  the  behavioral  approach 
to  simulation  is  that  the  primary  goal  of  all  simulators 
that  involve  human  participation  is  to  support  learning 
in  the  case  of  training  simulators  or  valid  system  per¬ 
formance  in  the  case  of  engineering  design  and  research 
simulators.  That  is,  the  purpose  of  a  simulator  is  to 
provide  the  conditions,  characteristics,  and  events 
present  in  the  operational  situation  necessary  for  the 
learning  of  skills  that  will  be  performed  with  actual 
equipment  or  for  eliciting  performance  identical  to  what 
would  occur  in  the  operational  context. 

Two  related  principles  derive  from  this  premise. 

First,  the  characteristics  and  methods  of  using  simulators 
should  be  based  on  their  behavioral  objectives.  Second, 
physical  realism  is  not  necessarily  the  only  or  optimal 
means  for  achieving  the  behavioral  objectives  of  simu¬ 
lation.  Because  the  history  of  simulator  development  is 
characterized  by  striving  for  improved  realism  through 
the  advancement  of  technology,  it  is  easy  to  forget  that 
the  learning  or  performance — not  physical  duplication — is 
the  primary  goal. 


Measures  of  Simulator  Effectiveness 

Following  World  War  II,  the  expense  associated  with 
the  development  of  simulators  became  of  increasing  con¬ 
cern.  As  early  as  1949  questions  about  the  importance  of 
physical  fidelity  of  simulators  were  raised  (Williams  and 
Flexman,  1949).  Robert  B.  Miller  (1954)  pointed  out 
that,  although  realism  and  cost  might  be  closely  related. 
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simulator  fidelity  and  value,  at  least  as  measured  by 
transfer  of  training,  probably  were  not.  There  is  a 
level  of  realism.  Miller  hypothesized,  at  which  the  cost 
of  additional  realism  rises  more  rapidly  than  the  addi¬ 
tional  training  value  that  can  be  derived.  Therefore, 
the  need  for  realism  in  all  simulators  began  to  come  into 
question.  Greater  use  could  be  made  of  simulation  if  the 
cost  of  simulators  could  be  reduced.  Studies  by  Semple 
et  al.  (1981b)  and  Waag  (1981)  on  visual  and  motion  simu¬ 
lators  indicated  that  physical  fidelity  is  not  essential 
in  evoking  transfer  of  training. 

A  basic  question  is  whether  simulators  can  be  as  effec¬ 
tive  as  actual  equipment  for  training  on  maintenance 
tasks.  Such  comparisons  were  made  in  12  different  studies 
and  summarized  by  Orlansky  and  String  (1977)  .  As  measured 
by  achievement  at  training  stations,  roughly  similar 
results  were  obtained  with  either  type  of  device.  Stu¬ 
dents  trained  with  simulators  performed  slightly  better 
in  two  cases  and  slightly  poorer  in  one  case.  These 
differences  were  statistically  significant  but  small. 

Another  study  on  the  extent  of  fidelity  required  was 
recently  performed  to  compare  the  maintenance  training 
effectiveness  of  actual  equipment  and  simulators  with 
both  high  and  low  physical  fidelity  for  the  6883 
Converter/Flight  Control  Test  Station  (Cicchinelli  et 
al.,  1980).  Maintenance  performance  test  scores  were 
essentially  equal  for  students  trained  on  the  actual 
equipment  and  on  either  simulator,  yet  there  are 
dramatic  differences  in  the  15-year  life-cycle  costs  for 
these  items.  Actual  equipment  cost  is  $5.3  million,  the 
high  physical  fidelity  simulator  cost  is  $2.1  million, 
and  the  low  physical  fidelity  simulator  cost  $1.6  million. 

Several  studies  involving  simulators  of  low  realism 
demonstrated  that  effective  training  could  be  conducted 
in  them  (e.g..  Prophet  and  Boyd,  1970).  In  fact,  for 
many  tasks,  training  in  low-realism  devices  enabled 
trainees  to  perform  as  well  on  operational  equipment  as 
personnel  who  were  trained  on  high-realism  devices  or 
even  on  the  operational  equipment  itself.  In  these 
studies,  the  low-realism  devices  had  training  value  equal 
to  that  of  high-realism  simulators. 

It  was  apparent  from  several  of  these  studies  that  the 
manner  in  which  the  low-realism  devices  were  used  was  a 
factor  in  the  effectiveness  of  the  training  conducted  in 
them.  Successful  training  concentrated  on  the  meaning, 
or  cue  value,  of  stimuli  present  in  the  device,  thus 
equating  functionally  rather  than  objectively  with  the 
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stimuli  and  responses  found  in  the  devices  and  in  the 
operational  equipment.  It  was  necessary,  of  course,  for 
trainees  to  conceptualize  realism  in  the  devices  to  make 
them  functionally  equivalent  to  simulators  of  much  higher 
physical  realism  compared  with  the  equipment  simulated. 
Given  these  requisites,  it  was  evident  that  the  high  cost 
of  realism  was  not  necessary  if  the  manner  in  which  a 
simulator  were  to  be  used  could  be  appropriately  struc¬ 
tured  along  behavioral  principles. 


Examples  of  Simulators  With  High  Fidelity 

As  noted  earlier,  pilots  now  may  be  certified  in  a 
simulator  that  meets  fairly  rigorous  fidelity  standards 
established  by  the  Federal  Aviation  Administration.  Such 
a  simulator  must  faithfully  duplicate  physical  and  func¬ 
tional  characteristics  of  an  aircraft  as  well  as  the 
conditions  of  flight.  Similarly,  in  engineering  design, 
where  critical  and  expensive  design  decisions  may  be 
based  on  performance  in  a  simulator,  high  fidelity  is  the 
best  insurance  for  obtaining  valid  performance  data. 

Specific  mission  rehearsal  is  another  instance  in 
which  high-fidelity  simulators  are  appropriate  for  train¬ 
ing.  The  objective  of  mission  rehearsal  is  to  attain  a 
final  performance  level  in  the  simulator  equivalent  to 
what  is  expected  in  the  operational  setting.  Mission 
rehearsal  involves  not  only  training  but  also,  implicitly, 
evaluation  of  performance  as  well.  High-fidelity  simula¬ 
tion  is  probably  necessary  to  satisfy  both  the  rigorous 
training  and  the  evaluation  components  involved. 

Thus,  while  low-realism  simulation  can  be  used  exten¬ 
sively  for  training  and  in  limited  ways  for  other  require¬ 
ments,  a  rational  case  can  easily  be  made  for  continued 
reliance  on  realistic  simulators  for  many  nontraining 
applications.  Even  in  the  case  of  training  simulators, 
there  are  likely  to  be  tasks  that  would  benefit  from 
realistic  simulation  because  of  the  difficulty  of  building 
training  exercises  that  would  permit  the  requisite  cue 
development.  In  addition,  personnel  attitudes  toward 
training  and  simulation,  particularly  when  familiarity 
with  mediational  processes  may  be  limited,  may  constitute 
justification  for  greater  realism  in  simulator  design. 
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THE  IMPORTANCE  OF  VALIDITY  IN  SIMULATOR  DESIGN 

Fidelity  in  the  sense  of  close  physical  correspondence, 
although  frequently  cited  as  a  measure  of  the  value  of  a 
simulator,  clearly  is  not  an  end  in  itself.  The  concept 
of  simulator  validity  must  be  considered.  This  term 
refers  to  the  correspondence  between  the  actual  results 
of  using  the  simulator  and  a  set  of  outcomes  that  are 
needed  or  desired  and  constitute  the  objectives  of  its 
use.  These  two  implications  of  simulator  validity  are 
discussed  below. 


Application  purpose 

To  evaluate  the  validity  of  a  simulator,  it  is  neces¬ 
sary  to  take  into  account  the  purposes  for  which  it  is  to 
be  used  and  to  formulate  objectives  for  that  use  (see 
Appendix  A).  Characterization  of  a  simulator  as  having 
high  or  low  validity  for  one  purpose  or  use  will  not 
necessarily  imply  the  same  degree  of  validity  for  another 
use.  Furthermore,  specific  objectives  are  required  in 
order  to  judge  whether  the  use  of  the  simulator  will  have 
outcomes  that  match  objectives  of  its  use.  For  example, 
training  as  a  general  purpose  is  insufficient  to  allow  a 
characterization  of  the  validity  of  a  simulator;  the 
specific  objectives  intended  to  be  accomplished  in  train¬ 
ing  with  the  simulator  must  be  considered. 

To  illustrate,  training  aids  consisting  of  two- 
dimensional  reduced-scale  paper  representations  of  air¬ 
craft  cockpits,  missile  control  panels,  and  other  opera¬ 
tional  equipment  have  been  used  very  effectively  for 
training  various  cognitive,  discriminative,  and  procedural 
tasks  associated  with  the  use  of  such  equipment.  There 
is  evidence  that  for  these  objectives,  outcomes  of  train¬ 
ing  are  quite  satisfactory;  that  is,  the  paper  devices, 
when  used  with  appropriate  mediational  processes,  have 
high  validity  for  the  purposes  of  training  cognitive, 
discriminative,  and  procedural  skills.  These  paper 
"simulators,"  however,  are  likely  have  low  validity  for 
training  or  assessment  of  manipulative  or  control  tasks, 
since  they  do  not  permit  these  components  of  performance 
to  be  practiced. 


Outcome  Specification 


To  evaluate  the  validity  of  a  simulator,  it  is  also 
necessary  to  have  a  set  of  outcome  specifications.  Using 
the  training  example,  outcome  specifications  consist  of 
the  performance  objectives  that  underlie  the  training 
programs  to  be  conducted  in  the  simulator.  For  personnel 
testing,  outcome  specifications  consist  of  the  performance 
that  will  be  sought  in  tests  that  will  use  the  simulator. 
For  use  in  system  design,  outcome  specifications  consist 
of  the  kinds  of  information  about  system  performance, 
including  human  performance,  that  will  be  assessed  in 
performance  trials  using  the  simulator. 

High  potential  validity  for  an  intended  outcome  indi¬ 
cates  that  the  outcome  can  be  fulfilled,  either  fully  or 
in  large  part,  provided  appropriate  programs  of  use  are 
carried  out.  Low  potential  validity  indicates  that  an 
intended  outcome  cannot  be  fulfilled  at  all  or  only  in 
small  part,  regardless  of  the  programs  in  which  the 
simulator  is  used.  In  the  case  of  a  simulator  to  be  used 
for  training,  its  level  of  potential  validity  can  be 
viewed  as  an  indication  of  the  amount  of  training  on  it 
that  potentially  can  transfer  to  the  operational  system 
or  equipment,  i.e.,  its  training  effectiveness  potential. 

Formulation  of  meaningful  outcome  specifications 
involving  behavioral  and  cognitive  factors  requires 
analysis  of  the  psychological  requirements  of  tasks  to  be 
performed  in  the  simulation  situation  as  well  as  in  the 
operational  environment  that  is  simulated.  The  need  for 
meaningful  outcome  specifications  implies  the  need  for 
procedures  or  processes  to  analyze  behavioral  and  cog¬ 
nitive  requirements  of  human  performance  in  developing 
specifications  for  simulator  systems. 

It  should  be  noted  that  these  behavioral  analyses 
cannot  be  reduced  to  a  uniform  procedure.  Knowledge  of 
learning,  cognition,  and  other  behavioral  processes  are 
necessary  to  understand  the  requirements  implied  by  the 
behavioral  outcomes  and  to  conduct  the  analyses  necessary 
to  develop  the  simulator  specifications. 
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CONTRIBUTIONS  OF  THE  BEHAVIORAL  SCIENCES  TO  SIMULATION 


The  key  to  successful  application  of  the  behavioral  1 

sciences  approach  to  simulator  design  and  use  is  broader  - 

and  more  detailed  understanding  of  the  behavioral  : 

processes  that  mediate  between  the  simulation  context  and 
elicited  performance.  The  development  and  refinement  of 
general  and  specific  behavioral  principles  applicable  to 
simulation  must  continue  to  advance  through  both 
laboratory-based  research  as  well  as  research  performed 

in  the  simulation  context  itself.  I 


REVIEW  OF  PREVIOUS  RESEARCH  RECOMMENDATIONS 

For  this  report  the  working  group  reviewed  research 
recommendations  drawn  from  a  variety  of  studies  performed 
to  assess  the  design  and  use  of  simulation.  Over  the 
past  30  years,  behaviorally  oriented  research  has  been 
stressed  frequently  and  consistently  as  a  critical  need 
to  improve  the  design  and  use  of  simulators.  Many  of  the 
findings  and  recommendations  are  formulated  in  general 
terms;  others  focus  on  specific  types  or  uses  of  simu¬ 
lators.  However,  even  in  these  latter  instances,  the 
generic  behavioral  questions  underlying  the  specific 
problems  are  apparent.  It  will  also  be  apparent  that 
many  of  the  same  issues  are  repeated  in  different  contexts 
or  have  persisted  in  the  same  contexts  for  decades. 

Most  of  the  reports  of  interest  are  from  the  literature 
on  flight  training  simulators.  This  reflects  the  fact 
that  flight  training  is  the  major  area  of  application  and 
has  the  longest  history  of  the  use  of  simulators,  involves 
the  greatest  number  and  most  expensive  devices,  and  there¬ 
fore  has  received  the  most  attention.  It  should  be  noted, 
however,  that  most  issues  that  have  been  identified  for 
flight  training  simulators  are  applicable  to  other  types 
and  uses  of  simulators  (see  Table  A-l  in  Appendix  A) . 

Table  5  shows  some  of  the  major  behavioral  research 
related  to  simulation  that  has  been  undertaken  in  the 
past  30  years.  The  research  is  organized  in  terms  of 
nine  major  topics,  which  are  discussed  below. 
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TABLE  5  Examples  of  Recommendations  for  Behavioral 
Research  Related  to  Simulation 

1.  Fundamental  Behavioral 
Processes 

Structure  and  acquisition 

of  skills  Gagne,  1954 

Muckier  et  al.,  1959 

Cognitive  skills  Prophet  et  al.,  1981 

Motivation  and  learning  Gagne,  1954 

Muckier  et  al.,  1959 

Behavioral  mechanisms 

of  transfer  Gagne,  1954 

Perceptual  learning  Hennessy  et  al.,  1980 

Visual  perception  National  Research 

Council,  1982 

2.  Fidelity  of  Simulation 

Important  and  unimportant 
simulator 

characteristics  Miller,  1954 

Effects  of  fidelity  on 

transfer  Muckier  et  al.,  1959 

Interaction  of  fidelity  with 

(a)  Instructional  variables 

(b)  Experience  level  Smode  and  Hall,  1966 

i  i  equ  i  i  «-im.  tit ::  Mv.C  I  wakey ,  1972 

Huff  and  Nagel,  1975 
Hayes,  1981 
Gaffney,  1981 

Departures  from  fidelity  of 
dynamics  to  compensate  for 

simulator  deficiencies  NATO-AGARD,  1980 

Adams,  1978 
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3.  Visual  Simulation 

Visual  display 
characteristics 


Muckier  et  al.,  1959 
Smode  and  Hall,  1966 
Huff  and  Nagel,  1975 
National  Research 
Council,  1975 
Hennessy  et  al.,  1980 
NATO-AGARD,  1980,  1981 
Kraft  et  al . ,  1981 


Scene  content  and 
visual  cues 


4.  Vehicle  Motion 
Motion  cues 


Interaction  of  motion 
and  vision 


Matheny,  1975 
National  Research 
Council,  1975,  1982 
Thorpe,  1978 
Hennessy  et  al.,  1980 
NATO-AGARD,  1980,  1981 
Prophet  et  al.,  1981 


Muckier,  et  al,  1959 
Huff  and  Nagel,  1975 
NATO-AGARD,  1980 
Prophet,  et  al,  1981 


Smode  and  Hall,  1966 
Matheny,  1975 
National  Research 
Council,  1975 


Interaction  of  motion  and 
skill  level 


Smode  and  Hall,  1966 


Effects  of  motion  on 
transfer 

5.  Performance  Assessment 


Smode  and  Hall,  1966 
Matheny,  1975 


Criteria  for  performance 


Gagne,  1954 
Center  for  Nuclear 
Studies,  1981 
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Performance  measurement 


Automated  performance 
monitoring 


Measurement  of  team 
performance 


6.  Modeling 


Gagne,  1954 
Muckier  et  al.,  1959 
National  Research 
Council,  1975 
USAF-SAB,  1978 
Center  for  Nuclear 
Studies,  1980 
Gaffney,  1981 
Prophet  et  al.,  1981 

USAF-SAB,  1978 
Center  for  Nuclear 
Studies,  1980 


Parsons,  1972 
Gaffney,  1981 
Prophet  et  al.,  1981 


Models  of  visual  and  motion 

simulation  Waag,  1981 

Sensory  system  modeling  USAF-SAB,  1978 

Model  of  multisensory 

spatial  orientation  NATO-AGARD,  1980 

Models  of  visual  environment 
to  identify  variables 

relevant  to  training  NATO-AGARD,  1980 

Models  to  predict  training 

effectiveness  Prophet  et  al.,  1981 

7.  Training 


Critical  characteristics 
for  transfer 


Effects  of  change  in  task 
characteristics  on 
transfer 

Measurement  of  transfer 
effectiveness 


Miller,  1954 
Caro,  1976 
Waag,  1978 


Muckier  et  al.,  1959 


Smode  and  Hall,  1966 
Williges  et  al.,  1973 
NATO-AGARD,  1980 


? 
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Systematic  method  for 
developing  training 
requirements  that 
provide  guidance  for 

simulator  design  AIAA,  1980 

NATO-AGARD,  1980 

8.  Training  Methods  Gagne,  1954 

HROL,  1953 
USAF  SAB,  1978 
Center  for  Nuclear 
Studies,  1980 

Sequence  of  training  McCluskey,  1972 

Use  of  feedback  and  guidance  Prophet  et  al.,  1981 

Instructional  features  Prophet  et  al.,  1981 

Instructor  training  Prophet  et  al.,  1981 

Simple  (part  task)  versus 
complex  (whole  task) 

simulators  Muckier  et  al.,  1959 

Smode  and  Hall,  1966 
National  Research 
Council,  1975 

Generic  versus  specific 

simulation  Center  for  Nuclear 

Studies,  1980 

9.  Other 

Documentation  and  use  of 
lessons  learned  from 

past  simulators  Smode  and  Hall,  1966 

Caro,  1976 

Experimental  design  Muckier  et  al.,  1959 

Williges  et  al.,  1973 


38 


Fundamental  Behavioral  Processes 

As  a  logical  extension  of  the  principle  that  simulator 
characteristics  should  be  determined  by  behavioral 
consequences,  several  studies  have  recommended  research 
on  the  behavioral  processes  relevant  to  simulation.  For 
example,  learning  or  skill  acquisition  is  the  fundamental 
process  of  concern  in  the  use  of  simulators  for  training. 
Gagne  (1954)  and  Muckier  et  al.  (1959)  advocate  research 
on  the  structure  of  skills,  the  acquisition  and  transfer 
of  learning  processes,  and  the  role  of  motivation  in 
learning.  Prophet  et  al.  (1981),  recognizing  that  the 
operation  of  most  modern  systems  increasingly  demands 
information  processing  and  decision  skills  as  much  or 
more  than  psychomotor  skills,  recommend  research  on  these 
and  other  cognitive  processes  as  a  basis  for  development 
of  training  procedures,  devices,  and  simulators  for 
cognitive  tasks. 

A  working  group  on  visual  simulation  of  the  National 
Research  Council's  Committee  on  Vision  concluded  that 
requirements  for  visual  scene  content  must  be  based  on  a 
better  understanding  of  perceptual  processes.  This  group 
recommended  research  on  a  computational  analysis  approach 
for  determining  visual  cue  requirements  and  verification 
through  psychophysical  experiments  as  well  as  research  on 
the  contributions  of  optical  flow  and  peripheral  vision 
to  spatial  orientation  (Richards  and  Dismukes,  1982). 
Hennessy  et  al.  (1980),  in  a  report  on  research  needs  for 
vertical/short  takeoff  and  landing  aircraft  simulator 
visual  systems,  proposed  several  studies  in  the  area  of 
perceptual  learning. 


Behavioral  Fidelity  of  Simulation 

Numerous  recommendations  have  been  made  to  investigate 
fidelity  of  simulator  characteristics  as  they  affect 
learning  or  performance.  Usually,  a  specific  domain, 
such  as  visual  cues  (Matheny,  1975;  National  Research 
Council,  1975;  NATO-AGARD,  1980b)  or  motion  cues  (Muckier 
et  al.,  1959;  NATO-AGARD,  1980b;  Prophet  et  al.,  1981), 
is  addressed.  Smode  and  Hall  (1966)  recommended  research 
on  fidelity  interactions  with  instructional  variables  and 
level  of  experience  of  trainees.  Other  recommendations 
on  simulator  fidelity  are  cast  more  broadly,  calling  for 
research  on  the  topic  in  general  (McCluskey,  1972;  Huff 
and  Nagel,  1975;  Hays,  1981;  NATO-AGARD,  1980b;  Gaffney, 


39 


1981).  In  all  instances,  the  authors  acknowledge  that 
fidelity  requirements  should  be  evaluated  in  terms  of  the 
behavioral  consequences  rather  than  in  terms  of  physical 
correspondence  to  a  real  system. 


Visual  Simulation 

Because  of  the  large  number  of  flight  simulators 
equipped  with  visual  systems  to  portray  outside  visual 
scenes  as  well  as  the  expense  of  these  systems,  there 
have  been  numerous  studies  in  this  area  to  identify  issues 
and  recommend  research.  Recommendations  fall  in  two  broad 
categories,  characteristics  of  visual  displays  and  scene 
content.  The  former  category  includes  recommendations 
for  determining  field  of  view  size  necessary,  requirements 
for  resolution,  luminance  and  contrast,  the  value  of 
color,  and  effects  of  distance  of  the  display  (Muckier  et 
al.,  1959;  Smode  et  al.,  1966;  Huff  and  Nagel,  1975; 
National  Research  Council,  1975;  Hennessy  et  al.,  1980; 
NATO-AGARD,  1980b;  Kraft  et  al.,  1980).  Many  of  the  same 
authors  and  groups  as  well  as  others  have  recommended 
research  on  scene  content.  Research  topics  include  scene 
detail,  degree  of  abstraction  of  portrayed  objects  and 
terrain,  texturing  of  surfaces,  the  kinds  and  density  of 
objects  in  the  scene  (Matheny,  1975;  National  Research 
Council,  1975,  1982;  Hennessy  et  al.,  1980;  NATO-AGARD, 
1980b,  1981;  Prophet  et  al.,  1981).  A  working  group  of 
the  Air  Force  Office  of  Scientific  Research  (Thorpe  et 
al.,  1978)  noted  that  there  is  not  even  an  adequate 
lexicon  for  describing  factors  in  scene  content. 

Issues  related  to  visual  simulation  have  been  with  us 
for  a  long  time.  Striking  advances  in  the  technology  of 
computer-generated  imaging  are  taking  place  now  that  will 
improve  visual  resolutions  and  their  quality  and  detail 
and  possibly  reduce  overall  cost.  It  is  possible  (some 
say  likely)  that  extremely  high  visual  fidelity,  in  the 
engineering  sense,  will  soon  overtake  the  current  need 
for  perceptual  research  on  visual  displays  for  simulators. 


Vehicle  Motion 

The  value  of  motion  cues  for  training  and  engineering 
design  flight  simulators  has  long  been  a  topic  of  debate 
and  one  of  the  few  areas,  along  with  visual  system 
requirements,  that  has  received  substantial  research 
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attention.  The  crux  of  the  problem  is  that  motion  plat¬ 
forms  and  other  motion  cueing  systems  cannot  duplicate 
sustained  accelerative  forces;  all  motion  cues  are  onset 
cues.  The  principal  issues  have  been  the  importance  of 
these  cues  for  training  and  performance,  the  form  of  the 
motion  functional  equations,  and  the  number  of  degrees  of 
freedom  (Muckier  et  al.,  1959;  Smode  et  al.,  1966;  Huff 
and  Nagel,  1975;  Matheny,  1975;  National  Research  Council, 
1975,  1980;  NATO-AGARD,  1980b,  1981;  Prophet  et  al., 

1981) .  Despite  the  large  number  of  studies  that  have 
been  performed  on  motion  cueing  (see  Waag,  1981),  there 
seems  to  be  no  consensus  on  its  value  (U.S.  Air  Force, 
1978) .  Waag  (1981)  attributes  this  state  of  affairs  to 
the  fact  that  the  studies  performed  were  so  specific  in 
nature  that  it  is  impossible  to  develop  any  general 
principles  or  conclusions  from  the  results. 


Performance  Assessment 

Performance  measurement  and  establishing  criteria  for 
performance  are  essential  not  only  to  all  uses  of  simu¬ 
lators  but  also  for  evaluating  and  improving  simulator 
design.  What  constitutes  necessary  and  effective  char¬ 
acteristics  of  simulators  remains  a  matter  of  opinion 
without  reliable  and  valid  performance  information.  For 
these  reasons  many  reports  on  simulation  have  recognized 
performance  assessment  as  a  key  area  for  research  (Gagne, 
1954;  Muckier  et  al.,  1959;  National  Research  Council, 
1975;  USAF  Scientific  Advisory  Board,  1978;  Center  for 
Nuclear  Studies,  1980;  Prophet  et  al.,  1981).  While 
there  are  a  host  of  problems  in  performance  measurement 
in  general  (these  are  discussed  in  Chapter  5),  the  devel¬ 
opment  of  automated  performance  measurement  systems  (U.S. 
Air  Force,  1978;  Center  for  Nuclear  Studies,  1980)  and 
the  measurement  of  team  performance  (Parsons,  1972; 
Gaffney,  1981;  Prophet  et  al.,  1981)  have  been  viewed  as 
particularly  pressing  problems  by  both  the  user  and  the 
research  communities. 


Modeling 

The  need  to  determine  simulator  requirements  based  on 
behavioral  research  is  reflected  in  the  recommendations 
presented  in  several  recent  reports.  The  USAF  Scientific 
Advisory  Board  (U.S.  Air  Force,  1978)  encouraged  continued 
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development  of  models  of  human  sensory  systems  that  can 
be  used  to  evaluate  the  consequences  of  motion  cueing.  A 
NATO-AGARD  group  (1980b)  made  similar  recommendations  for 
the  development  of  a  model  of  multisensory  spatial  orien¬ 
tation  and  a  model  of  the  visual  environment  to  identify 
variables  relevant  to  training.  Waag  (1981)  calls  for  a 
model  of  visual  and  motion  simulation  to  provide  a  struc¬ 
ture  to  aid  in  developing  testable  hypotheses  that  might 
subsequently  lead  to  generalizable  research  findings. 

Reflecting  many  of  the  same  concerns,  Prophet  et  al. 
(1981)  recommend  the  development  of  models  to  predict 
training  effectiveness  of  simulators  as  well  as  normative 
performance  models  for  proficiency  evaluation.  Several 
reports  that  have  included  recommendations  for  research 
on  behavioral  topics  relevant  to  simulation  have  also 
called  for  development  of  human  performance  models  in  the 
areas  of  vision  and  motion  sensation  and  perception  (Waag, 
1981;  U.S.  Air  Force,  1978;  NATO-AGARD,  1980b).  The 
importance  of  human  performance  models  to  simulator  design 
as  well  as  to  practical  application  of  simulators  is  dis¬ 
cussed  at  length  later  in  this  chapter. 


Training  and  Training  Methods 

Research  recommendations  specifically  related  to  the 
training  function  principally  address  issues  of  transfer 
of  learning  and  training  methods.  Answering  the  most 
fundamental  question — what  are  the  critical  simulator 
characteristics  necessary  to  produce  transfer — has  been 
discussed  as  a  research  need  (Miller,  1954;  Caro,  1977; 
Waag,  1981)  as  well  as  how  changes  in  task  characteris¬ 
tics  affect  transfer  (Muckier  et  al.,  1959).  Task  anal¬ 
ysis  and  instructional  system  development  methods  are 
useful  to  determine  what  is  to  be  trained  but  not 
explicitly  to  determine  what  is  required  to  conduct 
training.  In  recognition  of  this  shortcoming,  two  recent 
reports  include  recommendations  for  research  to  develop 
systematic  methods  for  establishing  training  requirements 
that  provide  guidance  for  simulator  design  (American 
Institute  of  Aeronautics  and  Astronautics,  1981; 
NATO-AGARD,  1980a) . 

In  acknowledgment  of  the  maxim  that  how  a  simulator  is 
used  is  as  important  as  its  physical  characteristics, 
several  reports  have  addressed  research  needs  to  improve 
the  methods  of  use  of  simulators  for  training  (Gagne, 
1954;  U.S.  Air  Force,  1978;  Center  for  Nuclear  Studies, 
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1980) .  Training  on  a  series  of  tasks  from  beginning  to 
end  as  they  will  be  performed  in  the  real  world  may  not 
be  the  most  effective  sequence  for  simulator -based  train¬ 
ing  and  has  been  recommended  as  a  subject  for  research 
(McCluskey,  1972)  .  Out  of  21  high-priority  research 
needs  to  improve  aircrew  training  devices,  developed  by 
Prophet  et  al.  (1981),  12  were  specific  to  simulator 
use.  The  research  topics  involving  utilization  included 
feedback  and  guidance  in  the  instructional  process,  con¬ 
trol  of  the  evaluation  and  measurement  process  for  indi¬ 
viduals  and  crews,  training  and  evaluation  of  instructors, 
and  self-instruction  methods. 

Although  simplified  or  part-task  simulators  have  a 
long  history  of  use  and  have  been  proliferating  since  the 
advent  of  inexpensive  microprocessors,  the  range  of  their 
potential  utility  is  not  clear.  Research  to  determine 
the  relative  merits  of  part-task  versus  whole  simulators 
for  training  and  what  tasks  are  most  suitable  for  each 
type  have  been  included  in  the  recommendations  of  several 
reports  (Muckier  et  al.,  1959;  Smode  et  al.,  1966; 

National  Research  Council,  1975)  .  A  closely  related 
issue  is  the  value  of  generic  training  devices  represen¬ 
tative  of  a  type  of  system  versus  simulators  of  specific 
systems.  This  is  a  particularly  important  issue  in  the 
training  of  nuclear  power  plant  operators  that  deserves 
research  attention  (Center  for  Nuclear  Studies,  1980). 


Documentation  and  Research  Design 

In  addition  to  the  research  described  in  the  above 
categories,  two  other  noteworthy  recommendations  have 
been  made  for  courses  of  action  to  improve  simulation 
that  have  been  drawn  from  the  literature.  Smode  et  al. 
(1966)  and  Caro  (1977)  have  advocated  documentation  of 
lessons  learned  from  past  and  existing  simulators  to 
improve  the  design  of  future  ones.  Implicitly  this 
suggestion  acknowledges  the  lack  of  a  systematic  base  of 
knowledge  about  simulator  design  practices.  Each 
technical  evolution  or  variation  in  simulator  design  is 
an  experiment  of  a  sort  and  worthy  of  assessment  and 
documentation  for  the  benefit  of  future  simulator 
developments. 

The  second  recommendation  is  for  both  research  and 
application  of  experimental  design  methodologies  that 
maximize  the  useful  information  that  can  be  extracted 
from  simulator-based  research  studies  and  engineering 
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test  and  evaluations  (Muckier  et  al.f  1959;  Williges  et 
al.,  1973).  Formal  experiments  tend  to  use  full  factorial 
designs  that,  for  practical  reasons,  limit  the  number  of 
variables  that  can  be  tested  in  one  study.  Engineering 
tests  and  evaluations  are  often  conducted  without  regard 
for  the  control  of  artifacts  or  the  possibility  of  formal 
statistical  analysis  of  the  derived  data.  Both  practices 
are  wasteful  in  terms  of  the  information  gained  for  the 
effort  expended.  The  need  for  further  development  and 
application  of  economical,  multifactor  designs  in  simula¬ 
tion  studies  is  discussed  in  Chapter  5. 


The  Persistence  of  Research  Needs 

All  these  previously  identified  issues  are  still  essen¬ 
tially  valid  questions  that  yet  have  to  be  answered.  The 
fact  that  the  same  research  issues  continue  to  persist 
suggests  that  the  required  research  has  not  been  performed 
or  the  research  that  has  been  performed  is  either  inade¬ 
quate  or  inconsequential. 

The  lack  of  basic  research  of  the  kind  necessary  to 
build  a  behavioral  base  of  knowledge  and  principles 
applicable  to  simulation  design  and  use  has  been  stated 
in  several  past  reports.  It  is  worth  repeating  their 
conclusions,  for  they  express  the  need  for  fundamental 
behavioral  research  as  pointedly  and  clearly  as  we 
possibly  could. 

Smode  et  al.  (1966)  reviewed  and  assessed  research 
relevant  to  pilot  training.  The  authors  preface  their 
specific  suggestions  for  research  needs  for  simulation 
training  with  the  observations  that  devices  used  earlier 
as  simulators  bear  little  resemblance  to  the  complex 
weapons  systems  trainers  and  training  facilities  in  use 
today.  However,  there  is  little  evidence  that  research 
has  been  performed  concerning  the  use  of  these  sophis¬ 
ticated  equipments.  What  research  has  been  undertaken 
appears  minuscule  considering  the  huge  stake  involved  in 
the  development  and  use  of  simulators. 

Smode  believes  that  the  real  issues  are  being  neglected 
and  that  a  serious,  sustained  program  is  needed  to  gain 
an  understanding  of  the  full  value  of  simulation  for 
aviation  training  and  for  the  specification  of  the  design 
requirements  necessary  for  achieving  the  potential  of 
simulation  training. 

What  research  has  been  done  has  contributed  little  to 
the  development  of  fundamental  principles.  Visual  and 
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motion  systems  for  flight  simulators  have  been  two  of  the 
most  intensively  researched  topics  in  simulation,  yet 
little  of  lasting  value  has  emerged.  Waag  (1981),  in 
concluding  a  review  of  the  literature  on  these  topics, 
noted  that  there  exist  no  quantifiable  models  of  visual 
and  motion  simulation  that  enable  testable  hypothesis  to 
be  generated  that  might  subsequently  lead  to  some 
generalizable  findings. 

The  National  Research  Council's  Committee  on  Vision 
addressed  research  needs  for  flight  simulation  and  also 
noted  the  lack  of  basic  research  in  this  area  (Richards 
and  Dismukes,  1982:8) : 

Current  research  attempting  to  define  relationships 
between  visual  simulation  variables  and  flight  con¬ 
trol  performance  has  immediate  value  for  evaluating 
specific  displays  and  equipment  features,  but  will 
not  provide  fundamental  knowledge  that  is  cumula¬ 
tive  and  that  might  allow  prediction  of  visual 
information  requirements  for  a  wide  range  of 
simulation  training  tasks. 

Huff  and  Nagel  (1975)  pointed  out  the  lack  of  develop¬ 
ment  of  fundamental  knowledge  of  behavior  relevant  to 
simulator  design  and  use.  They  indicated  that  it  is 
essential  that  new  ways  be  developed  to  conceptualize  and 
measure  pilot  experience  as  a  necessary  step  for  the  con¬ 
tinued  development  and  use  of  flight  simulators.  Current 
techniques  have  not  provided  a  firm  foundation  on  which 
to  build  simulation  design.  They  end  with  a  quotation 
from  Adams'  (1973)  preface  to  a  special  issue  of  the 
Journal  of  Human  Factors  devoted  to  simulation.  It  is 
equally  apt  to  quote  it  here,  more  than  a  decade  later 
(p.  501): 

We  often  seem  content  with  the  routine  testing  of 
training  devices  which  engineers  create,  and  this 
is  an  important  contribution,  but  where  are  our 
systematic  research  programs  that  will  produce 
principles  and  provide  guidance  for  the  simulation 
engineers  of  tomorrow? 


THE  NATURE  OF  THE  BEHAVIORAL  PROCESSES  IN  SIMULATION 

In  this  section  we  discuss  two  broad  topics:  analyses 
of  cognitive  processes  and  knowledge  and  human  performance 
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modeling.  We  believe  these  are  two  key  methods  for  char¬ 
acterizing  and  expressing  the  behavioral  processes  of 
interest  in  simulation  as  well  as  identifying  and  facili¬ 
tating  the  behavioral  research  necessary  to  improve  the 
design  and  use  of  simulators. 

Fundamental  to  realizing  the  behavioral  objectives  of 
simulation — the  desired  learning  or  performance — is  the 
need  to  understand  the  cognitive  processes  that  are 
responsible  for  the  interpretation,  transformation,  and 
retention  of  information,  its  consolidation  into  new 
processes,  i.e.,  skills  and  their  ultimate  manifestation 
in  performance.  From  a  behavioral  perspective,  the 
central  issue  is  understanding  the  internal  processing 
and  transformation  of  information,  the  development  of 
abilities,  internal  processes  themselves,  and  decision 
and  response  mechanisms  that  produce  overt  behaviors.  In 
other  words,  it  is  important  to  know  how  to  characterize 
the  hidden,  internal  cognitive  activities  and  how  external 
and  internal  events  interact. 

Complementing  the  need  for  analysis  of  cognitive 
processes  is  the  need  to  synthesize  these  processes  into 
a  coherent  and  tractable  form.  Human  performance  models 
fulfill  this  need  because  they  are  explicit  expressions 
of  behavioral  processes.  A  model  has  a  theoretical  base 
inherent  in  its  construction  and  is  both  extensible  and 
verifiable.  A  model  is  thus  an  orderly  means  to  accumu¬ 
late  behavioral  knowledge  and  predict  effects  that  can  be 
empirically  tested.  It  serves  as  a  framework  for  research 
programs  but  also  can  be  used  as  a  tool  for  the  design  of 
simulators  as  well  as  other  systems.  It  has  the  singular 
virtue  of  communicating  knowledge  of  behavior  in  the  same 
way  as  the  physical  models  that  design  engineers  are 
accustomed  to  using. 

Used  in  concert,  analyses  of  cognitive  processes  and 
knowledge  and  their  expression  in  human  performance  models 
provide  a  strong  mechanism  for  guiding  research  on  behav¬ 
ioral  issues  in  simulation  and  translating  the  research 
results  into  design  practice. 


Analyses  of  Cognitive  Processes  and  Knowledge 

Many  difficulties  in  the  design  and  use  of  simulators 
could  be  reduced  with  more  thorough  and  specific  analyses 
of  the  human  skills  and  knowledge  required  for  the  opera¬ 
tion  and  maintenance  of  equipment.  In  the  past,  contribu¬ 
tions  of  behavioral  scientists  to  simulation  have  been 
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based  on  the  available  psychological  principles  and 
methods  and  have  provided  useful  insights  into  the 
requirements  for  effective  training  and  other  functions 
using  simulation.  Recent  scientific  developments  have 
provided  methods  of  empirical  analysis  and  theoretical 
concepts  that  make  possible  a  significant  increase  in  the 
detail  with  which  cognitive  requirements  of  complex  task 
performance  can  be  specified. 

The  term  cognitive  science  is  used  to  refer  to  a 
growing  body  of  scientific  work,  using  methods  and  con¬ 
cepts  from  cognitive  psychology,  artificial  intelligence, 
and  linguistics,  consisting  of  analyses  of  relatively 
complex  cognitive  processes  and  systems.  Major  recent 
contributions  and  topics  of  rapidly  advancing  knowledge 
are  summarized  in  the  report  of  the  Research  Briefing 
Panel  on  Cognitive  Science  and  Artificial  Intelligence 
prepared  in  1983  for  the  President's  science  adviser 
under  the  auspices  of  the  National  Research  Council's 
Committee  on  Science,  Engineering,  and  Public  Policy 
(National  Academy  of  Sciences-National  Academy  of 
Engineering,  1983).  The  summary  given  here  emphasizes 
aspects  of  special  relevance  to  simulation. 


Skills  Acquisition  and  Instruction 

Scientific  analyses  of  the  cognitive  processes  and 
knowledge  that  produce  skilled  performance  provide  a  new 
target  for  the  analysis  of  learning.  With  more  detailed 
characterizations  of  the  knowledge  and  skills  that  are 
acquired,  analyses  of  learning  should  be  directed  toward 
understanding  how  these  specifically  characterized 
processes  are  learned. 

Cognitive  scientists  have  studied  the  process  of 
instruction  and  have  developed  analyses  of  the  cognitive 
processes  and  knowledge  used  in  successful  tutorial 
teaching.  Studying  performance  by  tutors,  Collins  and 
Stevens  (1981)  concluded  that  a  tutor  uses  a  model  of  the 
student  in  which  the  student's  knowledge  is  compared  with 
knowledge  of  the  subject,  represented  as  a  hierarchy  of 
topics  with  a  hierarchy  of  information  within  each  topic. 
Collins  and  Stevens  developed  a  model  that  simulates — and 
conducts — tutorial  instruction,  adding  knowledge  or 
changing  mistaken  beliefs  through  Socratic  dialogue. 

Another  idea  used  in  tutorial  systems  is  a  detailed 
model  of  the  system  that  a  student  is  learning  about, 
which  is  used  to  simulate  performance  of  that  system. 
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This  idea  is  used  in  the  STEAMER  system  (described  in 
Appendix  A) ,  which  simulates  the  operation  of  a  steam 
plant.  Students  are  thereby  able  to  observe  the  effects 
of  actions  they  can  perform,  such  as  opening  or  closing  a 
valve.  A  system  model  is  also  included  in  SOPHIE,  a 
tutorial  system  that  provides  instruction  in  trouble¬ 
shooting  electronic  equipment  (Brown  et  al.,  1983). 

SOPHIE  includes  a  simulation  of  a  piece  of  equipment  that 
enables  a  student  to  obtain  readings  of  voltage  or  current 
that  would  be  obtained  when  the  device  is  in  a  variety  of 
malfunctioning  states.  It  allows  the  student  to  specify 
faults  whose  effects  can  then  be  investigated.  SOPHIE 
also  includes  a  simulated  expert  troubleshooter  that  can 
identify  the  faults  that  student  selects,  allowing  the 
student  to  observe  successful  performance  in  the  trouble¬ 
shooting  task. 


Information  Processing 

Another  area  in  which  important  advances  are  occurring 
is  information  processing,  including  speech  understanding 
and  visual  information  processing.  A  new  theoretical 
advance  in  the  form  of  a  computational  theory  of  vision, 
developed  by  Marr  (1981) ,  makes  significant  headway  toward 
an  understanding  of  the  mechanisms  of  visual  information 
processing  that  convert  energy  patterns  on  the  retina 
into  information  about  contours,  surfaces,  textures, 
objects,  and  motion.  Visual  imagery,  including  imaged 
information  retrieved  from  memory,  has  been  studied  in 
some  detail  (Kosslyn  and  Schwartz,  1977),  and  it  has  been 
found  that  the  spatial  properties  of  images  (e.g.,  their 
sizes  and  the  relative  distances  among  their  parts)  play 
significant  roles  in  cognitive  tasks. 

The  relevance  of  information-processing  research  to 
simulation  includes  use  of  the  findings  to  guide  the 
design  of  information  presentation  in  simulators  and 
auxiliary  information  sources.  As  we  learn  more  about 
the  properties  of  information  in  visual  images,  we  can 
design  displays  to  present  the  information  critical  for 
the  cognitive  tasks  to  be  performed  and  the  skills  to  be 
acquired.  For  example,  the  use  of  the  computational 
theory  of  vision  to  determine  requirements  for  flight 
simulator  visual  systems  is  discussed  in  a  recent  report 
of  the  Committee  on  Vision  (Richards  and  Dismukes,  1982). 
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Qualitative  Understanding 

An  exciting  current  topic  of  research  in  cognitive 
science  is  the  investigation  of  cognitive  structures  that 
provide  qualitative  conceptual  understanding  of  procedures 
and  systems.  Although  instruction  in  science  and  engi¬ 
neering  emphasizes  formal  methods,  such  as  procedures  for 
calculation  and  the  correct  use  of  formulas,  it  is  widely 
understood  that  formal  knowledge  is  not  sufficient  for 
solving  problems  and  reasoning  about  novel  situations  in 
scientific  and  engineering  domains.  The  additional 
knowledge,  sometimes  called  "intuition,"  involves  under¬ 
standing  of  qualitative  relations  and  conceptual  struc¬ 
ture.  Recent  research  has  begun  to  identify  properties 
of  this  knowledge  and  to  develop  theoretical  methods  of 
characterizing  it  in  useful  ways. 

One  of  the  major  lines  of  research  on  qualitative 
understanding  involves  analysis  of  mental  models  that 
individuals  have  of  physical  systems  (Gentner  and  Stevens, 
1983).  In  general,  mental  models  consist  of  knowledge  or 
beliefs  about  causal  relations  among  components  of  sys¬ 
tems.  The  model  may  refer  directly  to  the  system  or  may 
be  based  on  analogy  with  some  other  system  that  the 
individual  knows  about  (or  has  beliefs  about) .  An  example 
of  the  latter  is  a  set  of  common  beliefs  about  electrical 
circuits,  based  on  an  analogy  to  hydraulic  systems. 

Another  line  of  research  is  investigating  understanding 
of  the  structure  of  causal  relations  among  quantities  in 
a  system.  For  example,  the  quantities  in  a  simple  elec¬ 
trical  circuit  are  linked  in  a  structure  of  causal  rela¬ 
tions:  the  total  amount  of  resistance  and  the  circuit 
voltage  determine  the  amount  of  current,  the  amounts  of 
resistance  connected  in  series  determine  the  total  amount 
of  resistance,  and  so  on.  Causal  knowledge  of  quantita¬ 
tive  structure  has  also  been  studied  in  mechanics,  where 
it  has  been  shown  to  provide  an  important  basis  for  the 
representations  that  expert  problem  solvers  construct 
(Larkin,  1983). 

As  more  is  learned  about  the  cognitive  structures  that 
constitute  general  conceptual  understanding,  the  design 
and  use  of  simulators  can  be  facilitated  by  identifying 
those  features  of  simulations  that  contribute  to  the 
acquisition  of  significant  qualitative  understanding  in 
training  and  that  permit  its  assessment  in  simulation- 
based  testing.  STEAMER,  mentioned  above,  is  one  example 
of  a  simulation  designed  to  communicate  general  causal 
knowledge  about  interactions  between  components  of  the 
system. 
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Human  Performance  Modeling 

As  person-machine  systems  become  more  and  more  complex, 
the  need  to  account  for  the  human  elements  in  these  sys¬ 
tems  in  an  orderly  and  quantitative  manner  assumes  ever- 
increasing  importance  and  recognition.  This  is  reflected 
not  only  in  the  growing  use  of  simulators  in  the  design 
and  evaluation  of  such  systems  but  also  in  the  expanding 
interest  in  the  modeling  of  human  performance.  The 
breadth  and  importance  of  human  performance  modeling  is 
such  that  a  separate  working  group  has  been  established 
by  the  Committee  on  Human  Factors  to  examine  this  subject 
in  detail.  In  this  report,  modeling  is  discussed  briefly 
in  terms  of  its  potential  roles  in  support  of  simulation 
studies  and  in  the  development  and  design  of  simulators. 

Human  performance  modeling  and  simulation  involving 
human  participants  are  complementary  activities  that  can 
be  used  iteratively  and  synergist ically  to  enhance  the 
achievement  of  overall  design  or  training  objectives. 
Appropriate  human  performance  models  can  play  a  role  prior 
to,  during,  ind  after  the  simulation  exercises  occur. 

This  role  is  apparent  when  one  considers  some  of  the 
basic  ;ntended  functions  of  human  performance  models  in 
person-machine  systems  analysis  (Pew  and  Baron,  1982): 

1.  To  provide  an  organized  approach  to  the  formu¬ 
lation  of  person/machine  problems  that  forces 
consideration  of  the  many  factors  influencing 
system  performance. 

2.  To  predict  "closed- loop"  person-machine  per¬ 
formance  . 

3.  To  provide  a  concise  and  systematic  framework 
for  organizing  experimental  data. 

4.  To  serve  as  an  embodiment  of  concepts  or 
derived  parameters  that  are  useful  as  measures 
of  performance. 


Engineering  Design  and  Research  Simulators 

As  has  been  noted  elsewhere,  the  development  of  engi¬ 
neering  requirements  for  human-in-the-loop  simulation  is 
a  complex  task  involving  numerous  trade-offs.  Among  the 
principal  issues  confronting  the  developer  of  a  simulator 
are  the  design  of  the  cue  environment  so  as  to  meet 
simulation  objectives  and  the  design  of  the  simulation 
model  of  the  functions  of  the  operational  system  to 
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fulfill  the  real-time  requirements  with  appropriate 
accuracy. 

In  specifying  the  cue  environment,  the  designer  must 
establish  the  need  for  particular  cues  as  well  as  the 
requisite  fidelity  for  their  presentation.  The  choices 
made  here  are  important  because  the  validity  and  utility 
of  the  resulting  simulation  can  be  critically  dependent 
on  them  and  because  the  decisions  involve  major  costs  of 
the  simulation.  Unfortunately,  these  decisions  are  quite 
difficult  to  arrive  at  rationally,  inasmuch  as  the  choices 
depend  on  complex  psychological  as  well  as  engineering 
factors.  The  specific  requirements  are,  of  course,  gov¬ 
erned  by  the  purpose  of  the  simulation;  training  simu¬ 
lators  have  different  needs  from  engineering  design  and 
research  simulators.  They  are  also  problem-dependent 
(e.g.,  the  need  for  motion  cues  in  the  analysis  of 
aircraft  control  in  a  gusty  environment  will  depend  on 
the  gust  response  of  the  aircraft) .  Finally,  the 
capabilities  of  the  adaptive  human  controller  both  help 
and  compound  the  problem.  The  human  operator  may  be  able 
to  compensate  for  simulator  shortcomings  and  maintain 
system  performance;  however,  this  could  result  in  negative 
transfer  in  a  training  environment,  or  reduced  acceptabil¬ 
ity  of  the  device,  or  an  incorrect  evaluation  in  a 
research  simulation. 

The  adequacy  of  the  cueing  environment  is  related  to 
the  accuracy  of  the  simulation  model  used  in  representing 
the  operational  system's  functional  characteristics,  as 
well  as  to  the  cue  generation  hardware  itself.  This  is 
particularly  true  for  discrete  simulation  models  that 
introduce  delays  in  the  information  presentation  paths 
that  are  present  in  the  actual  situation. 

In  the  development  of  engineering  design  and  research 
simulators,  the  consideration  of  a  suitable  model  will 
force  the  designer  or  researcher  to  a  more  precise  formu¬ 
lation  of  the  problem  to  be  investigated,  the  factors 
needing  control,  and  the  measures  of  human  and  system 
performance  that  are  appropriate.  Moreover,  with  a 
predictive  model,  it  is  possible  to  establish  and  evaluate 
preliminary  design  concepts  and  parameters.  Not  only 
will  these  activities  make  the  overall  evaluation  or 
experimental  process  more  efficient,  but  they  will  also 
maximize  the  likelihood  that  critical  conditions  will  be 
focused  on  in  the  simulation  environment. 

An  example  of  the  use  of  a  human  performance  model  for 
experimental  planning  is  given  in  the  work  of  Junker  and 
Levison  (1978) .  They  used  the  Optimal  Control  Model 
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(OCM)  of  the  human  operator  to  design  an  experiment  to 
examine  the  effects  of  platform  motion  cues  on  manual 
tracking  performance.  In  particular,  the  RMS  amplitude 
and  spectral  shape  of  the  system  inputs,  the  control 
gain,  and  the  performance  criterion  were  all  designed, 
using  the  OCM,  to  provide  conditions  that  would  resolve 
the  experimental  questions  while  meeting  the  experimental 
constraints . 

During  and  after  simulations,  models  can  serve  prin¬ 
cipally  as  aids  in  reducing,  interpreting,  and  extrapo¬ 
lating  the  data  obtained  in  the  simulations.  Because  of 
the  adaptive  nature  of  human  response,  changes  in  system 
variables  may  not  result  in  discernible  changes  in  system 
performance  under  the  conditions  investigated.  They  may, 
however,  result  in  subtle  alterations  of  human  response 
characteristics  or  behavior  that  may  be  important  in 
different  circumstances.  These  changes  are  often  more 
easily  discerned  in  terms  of  parameters  of  human  perfor¬ 
mance  models.  In  short,  model  parameters  can  be  succinct, 
diagnostic,  and  sensitive  measures  of  the  results  of 
simulator  evaluations  and  experiments.  Models  can  also 
serve  as  insightful  ways  of  looking  at  and  compressing 
empirical  data  so  that  they  can  be  extrapolated  to  new 
situations.  That  is,  predictive  models  can  be  used  to 
interpolate  or  extrapolate  to  conditions  that  are 
untested  in  the  course  of  a  specific  simulation  study. 

These  uses  of  human  performance  models  in  analyzing 
and  interpreting  simulator  data  are  widespread.  For 
example,  instrument  monitoring  models  have  been  used  as  a 
basis  for  analyzing  eye  movements  in  flight  simulators 
and  relating  those  movements  to  workload  (Senders  et  al., 
1969?  Weir  and  Klein,  1970)  .  An  instance  of  particular 
relevance  to  simulation  technology  is  provided  by  Levison 
and  Junker  (1977).  The  OCM  was  used  to  provide  a  single, 
coherent  explanation  of  simulator  data  on  the  effects  of 
motion  cues  on  performance  for  the  cases  of  disturbance 
motion  and  target  motion.  Phenomenological,  or  purely 
empirical,  descriptions  of  measured  human  and  system 
performance  yield  significant  differences  for  the  two 
motion  conditions.  However,  the  OCM  predicts  results 
without  changing  the  model  structure  or  human  behavioral 
parameters — i.e.,  it  predicts  how  the  human  uses  motion 
cues  differently  according  to  the  specifics  of  the  task. 
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Training  Simulators 

In  the  training  applications  of  simulators,  human 
performance  models  play  a  similar  role.  They  can  be  used 
prior  to  development  of  training  curricula  to  determine 
the  areas  of  human  performance  most  critical  to  knowledge 
of  the  system,  so  that  these  may  be  emphasized  in  the 
training  process.  Perhaps  a  more  important  contribution 
of  modeling  to  training  would  be  the  use  of  normative 
models  as  yardsticks  against  which  to  measure  trainee 
performance,  thereby  providing  mechanisms  to  be  used  in 
an  adaptive  training  scheme.  The  parameters  of  an 
analytic  model  may  also  prove  to  be  sensitive  measures  of 
operator  performance  and  learning.  These  uses  of  human 
performance  models  in  support  of  training  simulation  are 
principally  in  the  research  stage?  they  require  further 
development  in  modeling  skill  acquisition  and  in  methods 
for  integrating  such  models  into  the  instructional 
process. 


Plight  Simulators 

The  problems  in  simulator  design  involving  the  cue 
environment  and  real-time  functioning  have  received  the 
most  attention  in  the  context  of  flight  simulation. 
Recently,  the  potential  value  of  human  performance  models 
for  helping  to  resolve  them  has  been  recognized  and  sev¬ 
eral  research  and  development  efforts  have  been  under¬ 
taken.  One  of  the  early  efforts  was  aimed  at  developing 
a  model  for  human  sensory  mechanisms  that  woull  allow 
simulator  motion  cueing  systems  to  take  full  advantage  of 
their  basic  characteristics  (Borah  et  al.,  1977 j  . 
Individual  models  for  vestibular,  visual,  tactile,  and 
proprioceptive  sensors  were  combined  in  a  composite  model 
structure  that  used  a  Kalman  filter  to  integrate  the 
information  from  the  various  modalities.  Several  recent 
studies  have  centered  on  the  application  of  the  Opera¬ 
tional  Control  Model  (OCM) ,  as  this  model  allows  for 
direct  incorporation  of  sensory  submodels,  such  as  that 
just  mentioned,  to  provide  a  multicue  model  for 
continuous  manual  control. 

Baron  et  al.  (1980a)  developed  techniques  for  using 
the  OCM  to  predict  the  effects  on  performance  of  certain 
simulation  model  design  parameters,  such  as  integration 
scheme,  sample  rate,  data  hold  device,  etc.  The  model 
was  applied  to  a  relatively  simple  air-to-air  tracking 
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task  and  showed  significant  sensitivity  to  several  simu¬ 
lator  parameters.  Model  results  were  later  compared 
(Baron  and  Muralidharan,  1980)  with  data  from  an  experi¬ 
mental  study  of  Ashworth  et  al.  (1979),  and  the  agreement 
was  very  encouraging. 

In  a  series  of  studies  (Baron  et  al.,  1980b;  Baron, 
1981;  Ricard  et  al.,  1981),  the  OCM  was  used  to  examine 
human  performance  while  hovering  a  helicopter,  using  a 
computer-generated  image  visual  system  and  a  motion 
system  with  six  degrees  of  freedom.  The  hover  task  was 
separated  into  longitudinal  and  lateral  control  tasks. 
Performance/workload  effects  of  these  simulation  elements 
were  analyzed  by  incorporating  submodels  into  the  OCM. 

The  model  results  suggested  that  minor  simulator  defici¬ 
encies  could  result  in  substantial  performance  and/or 
workload  deviations  from  those  expected  in  actual 
flight.  Unfortunately,  there  were  no  corresponding 
experimental  data  to  confirm  or  deny  these  predictions. 

Also  of  interest  are  two  ongoing  experimental/ 
analytical  efforts  that  are  aimed  at  collecting 
additional  data  on  simulator  fidelity  while  further 
refining  the  pilot/simulator  model.  One  effort, 
sponsored  by  NASA-Langley  Research  Center ,  involves  both 
in-simulator  experiments  with  various  degrees  of  cueing 
(fixed  base,  platform  motion,  g-seat)  and  actual 
in-flight  duplications  of  the  same  flight  control  task. 
This  task  involves  a  high-performance  aircraft  (F-14) 
tracking  a  target  aircraft  that  is  executing  a  windup 
turn,  where  possible,  the  same  pilots  are  used  for  both 
in-simulator  and  flight  experiments.  Model  predictions 
will  be  used  in  the  experimental  design  phase.  These 
predictions  will  then  be  compared  with  the  data  obtained 
in  the  corresponding  simulator  experiments.  Parameters 
of  the  model  that  yield  the  best  match  to  the  data  will 
be  determined  to  tune  the  model.  These  parameters,  which 
will  also  reflect  the  individual  characteristics  of  the 
people  involved  in  the  experiment,  will  then  be  used  to 
predict  results  for  the  flight  experiment.  Again,  model/ 
data  comparisons  will  be  made  and  parameters  yielding  a 
best  match  to  the  flight  data  determined.  Discrepancies 
between  simulator  and  flight  data  will  be  explored,  using 
the  model,  to  provide  analytical  bases  for  the  differences 
in  terms  of  cueing  deficiencies  and/or  changes  in  pilot 
strategies  and  to  further  upgrade  and  develop  the 
analytical  models. 

The  second  effort  is  a  three-year  program  sponsored  by 
the  Air  Force  Aerospace  Medical  Research  Laboratory  to 
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develop  further  understanding  of  a  pilot's  use  of  visual 
and  motion  cues  so  as  to  aid  development  of  simulator 
requirements.  In  this  work/  a  significant  element  is  the 
development  of  models  for  the  processing  of  visual  stimuli 
to  generate  estimates  of  self-motion  (Zacharias  et  al., 
1983).  As  in  the  previously  described  work,  analytic 
models  are  being  used  before  simulation  for  experimental 
design  and  after  simulation  for  analysis. 

Models  of  human  performance  can  also  be  used  to  help 
design  algorithms  to  compensate  for  simulator  shortcom¬ 
ings.  For  example.  Crane  (1981)  has  used  the  crossover 
model  of  McRuer  et  al.  (1965)  to  determine  the  parameters 
of  a  lead-lag  network  to  be  used  to  compensate  for 
unwanted  time  delays  in  the  generation  of  visual  cues. 
Baron  et  al.  (1980a)  have  shown  how  the  OCM  could  be  used 
to  design  compensators  aimed  at  restoring,  as  closely  as 
possible,  pilot  performance  and  behavior  to  that  which 
would  be  obtained  if  there  were  no  simulator  delays.  The 
OCM  has  also  been  used  to  derive  a  washout  scheme  for  a 
motion  platform  (Baron  et  al.,  1980c). 

Finally,  it  should  be  mentioned  that  real-time  simu¬ 
lation  models  of  human  operators  may  be  valuable  as 
substitutes  for  some  of  the  operators  in  multiperson 
systems  evaluation,  research,  or  training.  Thus,  for 
example,  in  investigating  or  training  the  interactions  of 
the  crew  of  a  single  aircraft  in  dense  traffic,  simulating 
the  crews  of  other  aircraft  by  modeling  their  performance 
is  an  attractive  alternative  to  live  simulation  of  these 
crews  both  for  economic  reasons  and  as  a  means  for  reduc¬ 
ing  experimental  variability  or  maintaining  consistency 
in  the  training  environment. 


Research  Needs  for  Human  Performance  Modeling 

Of  course,  the  uses  of  models  described  above  in 
conjunction  with  simulation  presuppose  the  existence  of 
appropriate,  validated  models.  For  many  problems  of 
interest,  such  models  do  not  exist  or  have  not  been 
suitably  validated.  This  is  the  other  side  of  the  model/ 
simulation  synergism,  in  that  the  development  and  valida¬ 
tion  of  the  models  cannot  proceed  without  the  extensive 
input  of  data  from  experiments  conducted  with  simulators. 

Although  fairly  advanced  models  for  the  use  of  per¬ 
ceptual  cues  in  vehicle  control  exist,  there  are  still 
significant  gaps  in  the  knowledge.  For  example,  modeling 
the  use  of  instruments  and  of  motion  cues  (as  provided  by 
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motion  platforms)  is  well  advanced,  but  much  less  is 
known  about  external  visual  cueing  and  proprioceptive 
cueing.  In  addition,  the  models  are  generally  suited  to 
the  prediction  of  the  performance  of  skilled  operators; 
there  has  been  much  less  effort  and  progress  in  modeling 
the  effects  of  various  cues  on  skill  acquisition  or 
maintenance. 

The  use  of  models  to  guide  the  design  of  simulation  is 
likely  to  be  most  important  in  the  near  term  for 
simulations  in  which  vehicle  control  is  the  major  task. 
This  is  principally  because  the  models  for  the  operator 
are  most  highly  developed  for  this  task.  However,  as 
supervisory  control  models  that  treat  multicue,  multitask 
environments  are  developed,  similar  roles  are  envisioned 
for  these  models. 
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SPECIFIC  RESEARCH  ISSUES  IN  SIMULATION 


The  working  group  has  identified  nine  major  issues 
that  deserve  research  attention  or  other  action  to 
improve  the  design,  use,  and  application  of  simulators. 
These  issues  are  concerned  with  the  effective  employment 
of  simulators  that  either  have  not  been  previously  dis¬ 
cussed  as  issues  or  have  not  received  the  attention  they 
deserve. 

In  Chapter  4  we  noted  that  most  reports  on  behavioral 
issues  in  simulation  have  concentrated  on  research 
related  to  physical  fidelity  and  its  consequences  for 
performance  and  training,  general  behavioral  subjects 
(i.e.,  perception,  learning,  and  memory)  as  well  as 
verification  of  simulator  worth  (i.e.,  validity  and 
reliability  of  performance  and  transfer  of  learning). 
Support  requirements  for  operator  or  instructor  use  and 
training  features  have  also  been  identified  as  research 
topics,  but  have  not  been  emphasized  to  the  same  degree. 
Most  reports  have  focused  on  the  core  of  simulation  and 
not  on  the  supporting  features  that  make  a  simulator  a 
design  tool,  research  vehicle,  or  training  medium  rather 
than  simply  a  bare  representation  of  some  system. 

The  research  recommendations  proposed  in  previous 
reports  are  important  and  we  agree  that  they  should  be 
carried  out.  However,  we  do  not  wish  to  restate  and 
reargue  in  this  chapter  the  importance  of  the  recommenda¬ 
tions  made  in  the  past;  the  original  authors  do  that  well 
enough,  and  most  recommendations  have  been  repeated  often 
enough.  Instead  our  intent  is  to  focus  on  research 
issues  and  other  needs  that  are  new  or  have  not  received 
sufficient  emphasis,  are  relatively  independent  of  each 
other,  and  do  not  presume  major  advances  in  understanding 
the  fundamental  behavioral  processes  involved  in  simula¬ 
tion.  Each  of  the  specific  issues  is  a  problem  of  some 
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consequence,  and  the  recommended  research  and  actions 
have  significant  potential  for  improving  current  and 
future  applications  of  simulation. 

In  this  chapter  we  therefore  discuss  nine  specific 
issues  and  their  associated  recommendations,  which  fall 
roughly  into  three  categories:  design,  procedures,  and 
applications  of  simulators.  The  first  issue,  design 
guidelines  for  training  simulators,  addresses  the  problem 
of  collecting  behavioral  information,  gained  through 
research  as  well  as  past  experiences  with  simulators,  in 
a  form  that  is  useful  to  simulator  designers  and  managers 
of  simulator  procurement  programs.  The  second  issue, 
improving  task  analysis  methods,  focuses  on  the  need  to 
develop  a  more  formal  procedure  for  translating  opera¬ 
tional  performance  requirements  into  requirements  for 
simulator  characteristics  and,  in  the  case  of  training 
simulators,  into  methods  for  training.  The  third  issue, 
stress  in  simulator  training  and  testing,  addresses  the 
long-standing  concern  about  the  performance  and  learning 
differences  that  may  occur  between  any  simulated  and 
real-life  situation. 

The  next  three  issues  discussed  deal  with  efficient 
and  effective  use  of  simulators.  Scenario  construction 
and  simulator  control  features,  experimental  (test) 
design,  and  performance  measurement  are  all  key  components 
of  the  supporting  structure  of  simulation.  The  seventh 
issue,  guides  for  training  simulator  instructors, 
addresses  the  need  to  develop  the  means  for  educating 
instructors  in  the  capabilities  of  a  simulator  and  its 
proper  use;  doing  so  can  be  an  important  but  relatively 
straightforward  means  for  enhancing  the  value  of  training 
simulators. 

The  last  two  issues,  training  of  cognitive  skills  and 
training  for  failures  and  out-of-tolerance  conditions, 
focus  on  the  application  of  simulation  to  training 
operators  of  highly  automated  systems  in  which  the 
primary  tasks  will  be  assessing  abnormal  situations  and 
making  decisions  rather  than  performing  well-defined 
routine  procedures. 

The  discussions  of  the  nine  issues  follow  a  common 
format.  We  introduce  each  issue  with  a  statement  of  the 
problem  and  its  relevance  for  simulation.  We  then  present 
recommendations  for  research  or  other  courses  of  action 
and  state  the  potential  benefits  to  be  derived. 
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TRAINING  SIMULATOR  DESIGN  GUIDES 

To  the  extent  that  the  intended  use  of  a  simulator 
requires  that  it  have  a  close  physical  correspondence  to 
the  system  simulated,  simulator  design  criteria  are 
generally  obvious  and  straightforward.  To  the  extent 
that  intended  use  permits  a  simulator  to  deviate  from 
physical  correspondence  as  the  primary  design  criterion, 
or  as  technology  limitations  and  funding  constraints 
necessitate  such  deviation,  the  appropriate  design 
criteria  are  less  obvious.  In  the  case  of  a  training 
simulator,  for  example,  criteria  for  design  to  facilitate 
the  transfer  of  training  or  specification  of  stimuli  to 
be  provided  in  the  simulator  as  substitutes  for  stimuli 
found  in  the  operational  system  must  be  based  on  knowledge 
of  human  perception  and  principles  of  learning  as  well  as 
the  engineering  specialities  underlying  simulation  tech¬ 
nology.  Often,  simulator  design  teams  lack  the  requisite 
knowledge  of  perceptual  and  learning  concepts,  and  infor¬ 
mation  to  overcome  this  deficiency  is  not  readily  avail¬ 
able  to  them. 

An  additional  simulator  design  problem  relates  to  the 
simulator  controller  interface.  As  is  the  case  in  the 
design  of  any  complex  human-machine  interface,  the  con¬ 
troller  position  of  a  simulator  must  be  designed  around 
the  process  to  be  controlled.  In  the  case  of  a  training 
simulator,  that  process  is  the  instructional  process  and 
includes  problem  setup  and  control,  provision  of  guidance 
and  feedback  to  the  trainee  on-line  as  well  as  during 
debriefings,  and  measurement  of  trainee  performance  in 
real  time.  Such  processes  for  most  simulators  are  often 
poorly  defined,  and  easily  applied  criteria  for  controller 
position  design  seldom  exist.  In  fact,  the  inefficiencies 
in  the  use  of  many  simulators  can  be  attributed  to  the 
limitations  imposed  on  controllers  by  their  designs. 


Recommendations 

The  working  group  recommends  that  design  guides  be 
developed  for  use  by  personnel  involved  in  the  process  of 
specifying  requirements  for,  designing,  and  developing 
simulators  for  training.  Guides  are  needed  that  address 
functional  design  considerations,  the  cue  value  of 
stimuli,  and  the  extent  to  which  realism  can  be  concep¬ 
tualized  to  make  a  simulator  functionally  equivalent  to 
an  operational  system.  These  processes  include  operator- 
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controller  interactions,  the  provision  of  guidance  and 
feedback  to  operators,  the  measurement  and  meaningful 
summarization  of  performance  to  operators  and  controllers, 
and  the  development  and  administration  of  scenarios. 

These  guides  should  concentrate  on  the  definition  of 
the  objectives  for  and  processes  of  use  of  the  intended 
simulators.  The  guides  should  provide  detailed  instruc¬ 
tion  for  design  personnel  to  follow  concerning  identifica¬ 
tion  of  operator  tasks  to  be  performed  in  the  simulator 
and  cues  relevant  to  those  tasks  operationally;  descrip¬ 
tion  of  simulation  components  and  features  necessary  to 
the  practice  of  those  tasks  (including  cue  substitutions 
where  appropriate);  and  definition  of  the  procedures  to 
be  employed  by  the  simulator  controller  in  setting  up  the 
simulator  for  a  period  of  use  and  conducting  training, 
performance  measurement,  or  other  activities  in  it. 


Benefits 

The  benefits  to  be  derived  from  the  availability  of 
simulator  design  guides  are  primarily  reduced  reliance  on 
the  necessity  to  use  the  operational  system  as  the  prin¬ 
cipal  design  model,  with  a  consequent  reduction  in  simu¬ 
lator  complexity  and  cost,  and  the  more  efficient  and 
productive  use  of  simulators  to  achieve  the  purposes  for 
which  they  were  developed. 


IMPROVED  TASK  ANALYSIS  METHODOLOGY 

Improvements  are  needed  in  task  analytic  techniques 
and  associated  inferential  methodologies.  They  are  used 
to  estimate  attributes  and  parameters  of  the  user-system 
interface  and  personnel  subsystem.  The  use  of  behavioral 
science  and  technology  in  the  design  and  support  of  sys¬ 
tems  has  led  to  an  expansion  in  the  scope  of  applications, 
extension  into  more  detailed  levels  of  data,  and  the  need 
for  more  powerful  techniques. 

Task  analysis  was  introduced  originally  (Miller,  1953) 
for  the  purpose  of  developing  training  equipment  and 
programs  for  initial  cadres  while  systems  were  still  in 
engineering  design  and  production;  it  served  those  pur¬ 
poses  well.  The  major  uses  of  task  analysis  have  been 
expanded  into  other  applications  in  the  design  of 
user-system  interfaces,  the  development  of  operating 
procedures,  the  specification  of  technical  support 
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requirements,  the  assessment  of  imposed  workload,  and 
test  and  evaluation. 

Attempts  at  standardization  and  expansion  of  task 
analysis  in  the  form  of  Instructional  System  Development 
(ISD)  have  revealed  shortcomings  that  attenuate  the 
efficiency  and  validity  of  its  uses  (Cream  et  al.,  1978). 
The  managers  of  training  for  a  system  have  three  needs 
that  ISD  is  intended  to  address:  to  identify  needed 
training  and  training  devices;  to  design  curriculum, 
course  content,  and  instructional  strategy;  and  to 
provide  functional  requirements  and  designs  for  training 
devices  and  simulators. 

ISD  procedures  provide  a  systematic,  structured 
approach  to  generating  and  collating  task-descriptive 
data;  however,  these  procedures  provide  little  assistance 
in  making  integrative  design  decisions.  The  designer  and 
program  manager  are  left  to  their  own  intuitions  and 
experience  to  design  training  and  trainers. 


Recommendations 

A  long-term  research  program  should  be  initiated  to 
provide  tools  for  task  analysis  encompassing  the  descrip¬ 
tion  of  operational  tasks,  the  decomposition  of  task 
descriptions  into  parameters  and  attributes  relevant  to 
the  design  objectives,  and  synthesis  of  the  analytical 
data  into  requirements  or  specifications  for  system 
design.  The  research  should  address  the  following 
objectives: 

1.  Codification  of  the  Design  Objectives  and 
Decisions.  Design  objectives  and  decisions  to  which 
human  factors  analysis,  function  analysis,  task  analysis, 
and  test  and  evaluation  results  contribute  should  be 
identified  and  classified.  The  data  and  analyses  suf¬ 
ficient  to  provide  the  information  needed  to  make  design 
recommendations  should  then  be  specified  for  each  type  of 
decision.  Examples  of  these  objectives,  decisions,  and 
recommendations  are  requirements  for  information  display, 
training  content,  instructional  strategy,  and  test  plan 
objectives. 

2.  Identification  of  Information  Sources.  Sources 
should  be  identified  from  which  the  required  data  can  be 
obtained  as  well  as  the  data  or  attributes  of  tasks  that 
must  be  extracted  or  estimated  in  the  subsequent  analysis. 
Functional  description  cf  how  a  system  will  be  used 
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rather  than  engineering  design  documents  tends  to  provide 
more  useful  information  for  task  description.  This  type 
of  data  is  typically  found  in  doctrinal  statements,  mis¬ 
sion  concepts,  and  the  documentation  of  mission  analysis, 
engineering  simulation  tests,  and  failure  mode  and 
effects  analysis. 

3 .  Development  of  Explicit  Procedures  and  Algorithms 
for  Processing  the  Input  Task  Data.  Procedures  and 
algorithms  to  yield  the  parameters,  values,  and  estimates 
are  required  to  satisfy  the  design  objectives.  The  infer¬ 
ential  processes,  knowledge,  and  heuristics  of  qualified, 
successful  analysts  should  be  explored  as  a  source  of 
standardizable  procedures.  Emphasis  should  be  placed  on 
the  phases  of  synthesis  of  elementary  task  data  into 
design  requirements  and  specifications. 

4 .  Development  of  Automated  Processing  and  Aiding 
Techniques.  Automation  should  be  provided  for  compiling 
data  bases  and  files  on  task  descriptions,  analyzing  the 
task  descriptions,  managing  the  analysis  process,  and 
providing  "audit  trails"  for  specific  decisions  and 
products  made  during  the  process.  Automation  is  par¬ 
ticularly  necessary  for  these  functions  because  of  the 
numerous  changes  that  occur  during  the  system  design 
process  and  after  the  system  becomes  operational . 


Achievement  of  the  objectives  of  such  a  research 
program  will  significantly  increase  the  cost-effectiveness 
of  simulation  use.  The  validity  of  the  results  of  simula¬ 
tion  studies  and  their  use  for  training  will  be  increased. 
The  increased  efficiency  of  the  task  analysis  process  will 
reduce  the  work  hours  required,  and  the  explication  of 
analytic  procedures  will  reduce  the  levels  of  skill  and 
experience  required.  Finally,  the  products  of  the  task 
analysis  will  be  germane  to  the  design  objectives  and 
decisions  to  be  made. 


STRESS  IN  SIMULATOR  TRAINING  AND  TESTING 

Some  have  argued  that  simulators  are  not  appropriate 
vehicles  for  training  and  testing  because  they  do  not 
cause  the  same  levels  of  participant  stress  as  do  real 
operations.  In  airplanes  and  nuclear  power  plants,  the 
pilot  or  operator  may  feel  fear  or  arousal  because  of 
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what  is  actually  occurring.  The  same  emotions  may  not  be 
produced  in  a  simulator  because  there  is  no  physical 
risk.  The  assumption  underlying  such  objections  is  that 
the  stress  associated  with  operational  performance  affects 
what  is  learned  and  the  strength  of  the  skills  that 
develop,  particularly  those  involving  complex  decision¬ 
making  tasks.  That  is,  it  is  assumed  that  learning  is 
situationally  specific,  and  operational  performance  will 
be  inadequate  unless  the  learning  situation  includes 
exposure  to  real  stress. 

Psychological  stress  has  been  defined  as  an  unpleasant 
emotional  state  evoked  by  threatening  environmental 
events  or  stimuli  (Janis  and  Mann,  1977).  As  suggested 
by  the  Yerkes-Dodson  Law,  under  a  highly  aroused  emotional 
state,  normal  patterns  of  information  processing  are  upset 
and  both  performance  and  learning  deteriorate.  Using 
flight  as  an  example,  there  are  at  least  three  sources  of 
stress  that  may  be  identified.  The  first,  environmental 
stress,  involves  the  emotions  brought  on  by  the  flight 
itself  due  to  the  fear  of  injury  or  death,  the  fear  of 
height,  and  the  exhilaration  brought  on  by  the  motion 
stimuli  and  g  forces.  However,  most  experienced  pilots 
say  that,  except  for  early  in  one's  flying  career,  this 
source  of  stress  is  not  very  important  in  affecting 
learning,  piiot  performance,  or  decision  making. 

The  second  source  of  stress,  task  demand,  is  the  result 
of  the  attention  and  effort  (physical  and  cognitive  work¬ 
load)  needed  to  accomplish  the  flight  problem  itself. 

This  source  may  be  significant  for  certain  maneuvers  such 
as  instrument  approaches  and  aerobatics,  for  which  error 
tolerances  are  small,  but  for  most  flying  and  flight 
training  it  is  not  very  significant. 

The  third  source  of  stress,  goal  conflict  stress,  is 
due  to  what  Brecke  (1981)  has  called  the  conflict  between 
the  background  problem  and  the  flight  problem.  The 
background  problem  refers  to  nonflight-related  pressures 
to  make  flight  decisions  in  a  certain  way.  Such  factors 
as  peer  pressure,  family  pressure,  economics,  company 
goals,  and  commitment  to  prior  decisions  may  be  very 
important  sources  of  stress  when  placed  in  conflict  with 
an  operational  flight  problem.  The  level  of  stress  from 
the  conflicting  background  problem  depends  on  the  number 
and  importance  of  goals  that  may  go  unsatisfied,  the 
degree  of  commitment  to  adhere  to  some  course,  the  amount 
of  perceived  risk  (physical  and  emotional),  and  the  amount 
of  time  to  make  the  decision  (Janis  and  Mann,  1977:50) . 


63 


Recomroendat ions 

Although  the  stress  effects  occurring  in  real  environ¬ 
ments  are  purported  by  military  and  civilian  managers  to 
be  an  important  factor  in  realistic  training,  most  simu¬ 
lation  specialists  would  agree  that  there  exists  a  lack 
of  understanding  of  stress  that  occurs  or  can  be  induced 
in  simulators.  Very  little  research  has  addressed  prob¬ 
lems  in  this  area.  The  work  that  has  come  closest  to  this 
issue  are  the  studies  of  workload  and  divided  attention, 
although  none  has  been  performed  specifically  to  assess 
simulation  stress  effects. 

Research  needed  includes  a  review  of  the  clinical 
res  Teh  in  the  area  of  stress  and  an  analysis  of  what 
these  findings  suggest  about  the  control  of  vehicles  and 
process  plants.  It  should  also  include  empirical  inves¬ 
tigations  of  the  effects  of  stress  on  information  proces¬ 
sing,  decision  making,  and  vehicle  control  in  both  train¬ 
ing  and  testing.  Workload  research  is  being  conducted  at 
Virginia  Polytechnic  Institute  and  State  University  by 
Walter  Weirwille  and  Robert  Williges;  at  the  University 
of  Illinois  by  Chris  Wickens;  at  Wright- Pat ter son  Air 
Force  Base  by  Robert  O'Donnell;  and  at  NASA  Ames  Research 
Center  by  Sandra  Hart. 


Benefits 

The  ability  to  create  realistic  levels  of  stress  in 
simulators  would  go  a  long  way  in  convincing  regulators, 
managers,  and  experienced  operators  of  the  validity  of 
performance  exhibited  in  simulators.  However  useful  this 
may  be,  it  is  essential  to  first  determine  whether  it  is 
important  to  induce  stress  in  some  or  all  simulation 
applications.  Failure  to  do  so  will  simply  relegate 
stress  to  the  same  status  as  physical  fidelity,  i.e., 
uncritically  accepted  as  an  essential  requirement. 


SCENARIO  CONSTRUCTION 
AND  SIMULATION  CONTROL  SUPPORT  FEATURES 

War  games  and  command-and-control  simulations  are 
somewhat  impeded  by  the  difficulty  of  changing  scenarios 
or  scenario  elements.  Process  and  vehicle  simulations 
could  be  used  more  effectively  and  efficiently  if  the 
actions  of  the  participants  were  less  free  and  more 
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subject  to  the  intentions  of  the  simulation  controller 
imposed  through  a  structured  and  easily  manipulated 
scenario.  Current  methods  for  generating  and  controlling 
scenarios  are  deficient  and  overly  burdensome  on  the 
simulator  controller.  There  are  three  areas  in  which 
simulation  controllers  need  assistance:  (1)  constructing 
and  compiling  scenarios  adequate  to  achieve  the  objec¬ 
tives  of  the  simulation,  (2)  running  the  scenario, 
compiling  the  required  performance  data,  and  monitoring 
the  simulated  events  for  compliance  with  the  objectives 
and  plan,  and  (3)  managing  and  adapting  the  scenario  and 
simulated  exercises  in  real  time  to  enhance  effectiveness. 
This  last  capability  is  especially  important  in  training 
when  the  instructor  can  use  knowledge  of  the  learning 
process  and  the  student’s  status  to  select  exercises  and 
variants  of  conditions  that  will  maximize  the  student's 
learning. 

One  might  wish,  for  example,  to  test  the  robustness  of 
a  particular  finding  by  running  the  same  problem  with 
variations  of  some  parameters  and  collateral  variables. 

Or  one  might  wish  to  test  the  generalizability  of  a 
particular  decision  process  by  running  problems  amenable 
to  the  same  process  but  involving  different  situation- 
specific  details.  Such  flexibility  is  currently  limited 
to  minor  variations  on  standardized  scenarios. 

Designing  and  compiling  scenarios  suitable  to  specific 
simulation  objectives  and  loading  these  scenarios  into 
the  computer  are  very  time-consuming  and  not  well- 
structured  activities,  and  therefore  must  be  done  by 
persons  who  are  skilled  or  knowledgeable  in  the  relevant 
technical  area,  software  techniques,  and  the  methodology 
of  simulation.  Procedures  and  aiding  techniques  analogous 
to  authoring  languages  are  needed  to  simplify,  organize, 
and  make  the  activities  of  scenario  generation  and  simula¬ 
tion  control  more  manageable,  flexible,  and  efficient. 


Recommendations 

A  program  of  research  projects  should  be  undertaken 
with  the  objectives  of  automating  parts  of  the  processes 
of  generation,  modification,  and  management  of  scenarios 
and  providing  simulation  controllers  with  computer  aids 
to  enhance  their  ability  to  perform  these  activities. 
These  projects  should  encompass  the  following  topics: 
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1.  Description  of  the  Activities  and  Processes  of 
Scenario  Generation  and  Simulation  Control.  The  activi¬ 
ties  of  scenario  generation  and  simulation  control  should 
be  described  at  several  levels  of  detail  to  permit  under¬ 
standing  of  the  functional  flow  of  information,  the 
identification  of  the  serial  and  parallel  sequences  of 
decisions,  the  actions  and  discriminations  required  of 
the  controller,  and  the  knowledge  and  inferences  on  which 
choices  and  action  depend.  The  activities  described  will 
be  allocated  to  controller,  computer,  and  supporting  aids. 

Description  of  these  activities  will  start  with  the 
statement  of  simulation  objectives  by  a  user  organization 
and  the  subsequent  translation  of  those  objectives  into  a 
sufficient  set  of  scenario  requirements.  These  require¬ 
ments  will  include  elements  such  as  organizations,  plat¬ 
forms  and  models  needed,  events  and  conditions  required 
to  provide  data  relevant  to  the  simulation  objectives, 
initial  conditions,  exercises,  data  collection,  and  data 
analysis.  They  should  also  include  procedures  for  con¬ 
verting  simulation  or  training  objectives  into  strategies 
for  accomplishing  these  objectives.  These  requirements 
in  turn  should  be  converted  into  a  process  for  retrieving 
the  necessary  model  and  algorithms  from  a  library  and 
compiling  them  into  the  software  for  the  simulated  exer¬ 
cises. 

2.  Development  of  Explicitly  Stated  Human-Machine 
Procedures  for  Operating  a  Simulation.  These  procedures 
should  include  generating  scenario  requirements,  compiling 
scenarios  and  exercises,  conducting  exercises,  modifying 
exercises,  keeping  track  of  key  events  and  participant 
behaviors,  and  analyzing  or  interpreting  the  results  in 
relation  to  the  objectives  of  the  simulation.  The  inter¬ 
face  between  the  controller  and  computer  should  be  user- 
friendly;  the  user-system  dialogue  should  be  based  on  the 
terminology  characteristic  of  the  technical  domain  of  the 
simulation  and  should  not  assume  a  more-than-average 
knowledge  or  understanding  of  computer  technology. 

3.  Development  of  Computer  Aids  to  Support  the  Con¬ 
troller  in  Scenario  Generation  and  simulation  Control. 

The  descriptions  of  the  processes  of  scenario  generation 
and  simulation  control  should  be  revised  to  identify 
activities  that  can  be  simplified  or  reduced  in  difficulty 
by  automation  or  supplying  aids  to  assist  the  controller, 
increase  efficiency,  and  increase  the  effective  use  of 
simulation  resources. 

4 .  Development  of  Simulation  Controller  Training 
Programs  and  Training  Techniques.  These  programs  should 
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emphasize  not  only  the  operation  of  the  simulator  but 
also  the  purposes  of  simulation  in  design,  testing,  and 
training.  The  underlying  behavioral  processes  should  be 
taught;  use  of  the  simulation  relative  to  the  effect  of 
training  on  these  behavioral  processes  should  be  covered. 
The  need  for  this  training  is  based  on  the  premise  that, 
although  some  operational  or  engineering  knowledge  may  be 
necessary  for  the  controller  position,  it  is  not  suffici¬ 
ent  to  provide  adequate  and  efficient  use  of  the 
simulator . 


Benefits 

Even  moderate  progress  on  this  project  can  greatly 
increase  the  value  returned  on  the  money  invested  in  the 
development  and  operation  of  simulation  and  simulators. 
Simulators  are  costly  to  operate  even  though  they  may  be 
less  expensive  than  their  alternatives;  in  some  cases 
alternative  means  do  not  exist.  However,  we  fail  to 
retrieve  the  data  and  obtain  the  information  inherent  in 
the  capabilities  of  the  simulator  and  its  data  processing 
resources.  Operating  procedures  are  also  inefficient, 
relying  too  frequently  on  manual  procedures  for  setup, 
operation,  and  analysis.  This  area  of  research  can  pay 
off  in  terms  of  greater  capability,  lower  operating 
costs,  more  effective  utilization,  and  higher  levels  of 
operational  proficiency. 


EXPERIMENTAL  DESIGN 

Simulation  is  increasingly  used  to  assess  design  and 
performance  factors  for  large,  complex  systems  that  often 
involve  many  interacting  human  operators.  Examples  of 
these  systems  are  air  traffic  control  and  various  command, 
control,  and  communication  systems.  The  scale  required 
for  these  simulations  aggravates  existing  problems  and 
creates  new  ones.  Issues  arising  in  all  phases  of  such 
simulations  include  simulation  features  and  fidelity 
requirements,  data  collection  and  analysis  procedures, 
and  experimental  design  and  test  structuring.  The  simu¬ 
lation  features  and  fidelity  problems  are  the  typical 
ones  of  balancing  realism,  comprehensiveness,  and  cost. 

For  example,  how  can  realistic  scenarios  be  generated  and 
implemented?  Must  all  subsystems  be  simulated?  Must 
humans  play  all  supporting  roles  or  can  computer-based 
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models  be  used  instead?  In  the  data  collection  and 
analysis  area,  the  questions  center  on  the  choice  of 
appropriate  performance  measures  and  data  analysis 
methods. 

Experimental  and  test  design  concerns  the  structuring 
of  simulator  characteristics,  events,  and  conditions 
efficiently  to  acquire  human  performance  information  that 
is  valid  and  reliable.  Important  objectives  of  experi¬ 
mental  and  test  design  are  to  allow  inclusion  of  the 
important  variables,  control  extraneous  factors  that  may 
affect  performance,  and  facilitate  analysis  and  inter¬ 
pretation  of  performance  data. 

The  major  problems  confronting  systems  research  and 
development  are  the  need  to  investigate  simultaneously 
several  relevant  factors  (often  at  several  levels)  and 
the  difficulties  in  maintaining  control  of  the  many 
factors  (both  those  to  be  varied  and  those  to  be  "fixed") . 
The  existence  of  a  large  number  of  interacting  variables 
often  makes  it  economically  impractical  to  perform  a 
complete  crossed  factorial  design,  i.e.,  every  level  of 
each  factor  combined  with  every  level  of  all  other 
factors. 


Recommendations 

The  working  group  recommends  three  general  actions  to 
enhance  the  use  of  sophisticated  experimental  design 
methods  in  the  conduct  of  simulation-based  engineering 
tests  and  evaluations  of  complex  systems:  greater  use  of 
existing  economic,  multifactor  techniques,  training  of 
more  people  in  the  use  of  these  techniques,  and  research 
to  develop  additional  procedures  for  designing  experiments 
and  tests  for  the  complex  situations  that  arise  in  the 
system  development  process. 

1.  Applications  of  Existing  Techniques.  There  are 
existing  economical  experimental  design  procedures, 
although  rarely  used,  that  are  quite  appropriate  for 
simulation-based  research  and  engineering  test  applica¬ 
tions.  For  example,  Simon  (1977)  and  Diamond  (1981) 
summarize  the  use  of  procedures  based  on  fractional- 
factorial  designs  as  efficient  means  for  screening  a 
large  set  of  independent  variables  to  provide  a  smaller 
set  of  the  most  important,  relevant  variables  for  sub¬ 
sequent  experimentation.  Box  et  al.  (1976)  provide  a 
detailed  discussion  of  the  use  of  complex  experimental 
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design  procedures  such  as  central-composite  design  to 
facilitate  the  development  of  empirical  models,  which 
can,  in  turn,  be  used  to  predict  human  performance  in 
complex  systems.  Recently,  Williges  and  Mills  (1982) 
discussed  several  of  the  experimental  design  and 
methodological  issues  related  to  the  use  of  simulation  in 
the  conduct  of  complex  systems  experimentation. 

Great  improvement  in  both  human  factors  research  and 
engineering  development  could  be  realized  through  the 
application  of  these  sophisticated  experimental  design 
procedures  in  simulation  environments.  Designs  dealing 
with  procedures  for  screening  independent  variables, 
economical  selection  of  data  points,  and  efficient, 
simultaneous  investigation  of  multiple  factors  are 
particularly  appropriate  for  application  in  simulation- 
based  test  and  evaluation  of  systems. 

2.  Skills  in  Using  Experimental  Designs.  Often 
complex  experimental  design  procedures  are  not  used  in 
simulation-based  test  and  evaluation  because  these  pro¬ 
cedures  are  not  well  known  or  understood  by  human  factors 
professionals  and  are  even  less  familiar  to  the  engineers 
or  systems  analysts  who  may  be  responsible  for  conducting 
the  tests.  There  is  a  clear  need  to  institute  graduate 
level  courses,  workshops,  and  continuing  education  semi¬ 
nars  on  advanced  experimental  design  procedures  for  human 
factors  applications. 

In  addition  to  education  there  is  also  a  need  for 
automated  experimental  design  aiding.  The  availability 
of  interactive,  computer-based  methods  for  the  development 
of  various  design  alternatives  would  make  many  of  the 
complex  experimental  design  procedures  more  readily 
accessible  and  more  easily  implemented.  Some  prototype 
experimental  design  aiding  systems  (System  Development 
Corporation,  1980,  1981)  have  been  developed  for  con¬ 
ducting  screening  studies  using  fractional  factorials. 
However,  additional  work  is  needed  to  develop  a  complete 
set  of  automated  design  aids  along  the  lines  of  other 
expert  aiding  systems. 

3.  Develop  New  Research  Procedures.  A  by-product  of 
encouraging  more  applications  of  advanced  experimental 
design  procedures  to  simulation  environments  will  be  the 
emergence  of  specific  needs  for  new  design  procedures. 
These  requirements  will  stimulate  methodological  research 
and  lead  to  the  development  of  new  and  improved  research 
techniques. 

One  research  need,  for  example,  is  a  more  detailed 
procedure  for  conducting  sequential  experimental  designs. 
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Due  to  the  extreme  complexity  of  simulation  environments, 
even  the  most  economical  experimental  design  is  not  suf¬ 
ficient  to  provide  data  on  all  human  factors  interface 
and  system  design  questions.  Few  procedures  currently 
exist  for  developing  sophisticated  strategies  for  con¬ 
ducting  a  sequential  set  of  interrelated  behavioral 
experiments  in  an  efficient  manner.  Methodological 
research  is  needed  to  develop  these  procedures  and  test 
them  in  simulation  environments. 


Benefits 

The  main  benefit  of  employing  economic  multifactor 
design  techniques  will  be  an  increase  in  the  efficiency 
and  effectiveness  of  using  simulators  as  means  for  inves¬ 
tigating  human-machine  systems  at  the  necessary  level  of 
complexity.  The  results  of  this  work  are  likely  to  lead 
to  the  development  of  general,  usable  design  principles. 
Likewise,  increased  application  and  development  of 
advanced  experimental  design  procedures  also  will  improve 
the  sophistication  and  timeliness  of  human  factors  con¬ 
tributions  to  the  systems  development  process. 


PERFORMANCE  MEASUREMENT 

Human-system  performance  is  measured  in  simulators  to 
provide  information  to  aid  decisions  for  system  design, 
personnel  selection,  training,  certification,  and  human 
performance  research.  Each  use  of  simulation  requires 
both  common  and  unique  performance  measures. 

Measures  for  system  design  should  provide  information 
about  human  capabilities  and  limits  and  predict  human- 
system  performance  during  envisioned  operations  for  given 
system  configurations.  Measures  for  job  sample  simulation 
should  augment  other  personnel  selection  test  data  and 
provide  information  to  predict  success  in  training  and 
actual  job  performance.  Measures  for  training  should 
yield  information  about  the  learning  process,  diagnose 
performance  deficiencies,  provide  feedback  to  the  student 
and  the  instructor,  aid  decisions  for  training  prescrip¬ 
tion  and  management,  and  predict  operational  performance. 
Measures  for  certification  should  yield  information  on 
whether  an  operator  or  maintainer  can  perform  all  routine 
and  emergency  tasks  to  specified  levels  of  proficiency 
and  safety.  Measures  for  research  should  provide  informa- 
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tion  to  elucidate  cognitive,  perceptual,  or  response 
processes. 

Performance  measurement,  while  a  problem  in  many 
settings  (e.g.,  on  the  job,  in  training  schools,  and  in 
behavioral  research  laboratories)  is  particularly  relevant 
in  simulation  applications  because  of  the  task  complexity 
and  the  frequent  need  for  real-time,  automated  performance 
measurement  in  most  instances.  Although  automated  mea¬ 
surement  subsystems  are  being  specified  as  a  requirement 
of  modern  simulators,  there  are  no  universal  or  fully 
validated  measurement  methods  or  measures.  The  need  for 
improved  human  performance  measurement  methods  is  well 
documented  (Muckier  et  al.,  1959;  Pope  and  Meister,  1977). 
A  recent,  large-scale  study  of  aircrew  simulator  training 
effectiveness  (Semple,  1981)  concluded  that  most  existing 
automated  training  performance  measurement  systems  are  so 
poorly  designed  as  to  be  useless. 

The  principal  factors  impeding  development  of  simulator 
performance  measurement  systems  are:  (1)  user  information 
requirements  are  not  always  defined  to  the  degree  of 
precision  necessary,  (2)  performance  criteria  are  often 
vague,  (3)  documentation  to  guide  measurement  system 
design  is  deficient,  and  (4)  past  measurement  system 
designs  have  been  oversimplified  and  are  an  inadequate 
base  for  new  developments. 

For  most  uses  of  simulation  a  good  start  on  the 
definition  of  what  to  measure  can  be  derived  from  a 
front-end  analysis  to  establish  the  performance  objectives 
and  standards  of  each  task.  For  procedural  tasks  for 
which  there  are  a  limited  number  of  correct  ways  to  do 
the  task  and  the  performance  standards  are  well  known  and 
precise,  a  front-end  analysis  can  yield  a  complete  speci¬ 
fication  of  what  to  measure. 

As  the  task  domain  changes  from  simple  fixed  procedures 
requiring  many  specific  overt  actions  to  complex  dynamic, 
interactive,  and  cognitive  tasks  for  which  the  proper 
action  is  not  entirely  defined,  what  to  measure  tends  to 
become  more  speculative.  Consequently,  measurement 
systems  may  include  measures  that  are  not  certain  to  be 
useful;  that  is,  researchers  and  test  directors  often 
record  everything  possible.  Deciding  which  measures  are 
important  and  how  to  combine  them  into  composite  perfor¬ 
mance  scores  or  figures  of  merit  is  frequently  deferred 
to  the  point  at  which  the  data  are  reduced  and  analyzed. 
This  process  commonly  involves  cleaning  up  the  data  by 
removing  unwanted  samples  after  examination,  computing  a 
variety  of  potentially  interesting  measures  such  as  error 
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in  terms  of  deviation  from  a  desired  profile  or  sequence 
of  tasks,  and  performing  several  statistical  analyses. 
Simple  measures  showing  large  differences  in  performance 
usually  receive  more  attention  than  more  complex  and 
subtle  indicators  of  performance.  Typically,  only  data 
recording  occurs  in  real  time.  The  remainder  of  the 
measurement  and  analysis  process  requires  days  to  months 
to  complete. 

For  a  real-time  automated  performance  measurement 
system,  however,  all  the  judgments  that  enter  into  (1) 
determining  when  tasks  start  and  end  for  measurement 
purposes,  (2)  cleaning  up  the  data  (e.g.  removing  false 
starts  or  selecting  data  on  outside  criteria),  (3) 
selecting  measures  of  interest,  and  (4)  weighting  several 
measures  to  develop  composite  figures  of  merit  have  to  be 
made  ahead  of  time  and  implemented  as  algorithms  that 
control  a  real-time  measurement  system. 

If  automated  measurement  systems  are  to  provide  data 
to  assist  the  judgments  of  qualified  instructors  or 
evaluators,  these  systems  must  incorporate  some  of  the 
knowledge  and  evaluation  rules  and  assume  they  are  validly 
used  by  humans  who  currently  perform  these  functions  and 
provide  feedback  to  control  training.  Moreover,  real-time 
measurement  systems  may  have  to  scale  the  quality  of  per¬ 
formance  in  operational  terms,  such  as  the  probability  of 
mission  success  for  design  studies  or  a  grade  for 
training. 

In  short,  there  are  human-system  performance  measure¬ 
ment  problems  that  have  been  known  for  a  long  time  but 
only  partially  solved.  Some  of  the  issues  are  development 
of  operationally  meaningful  performance  measures  and 
criteria,  implementation  of  some  of  the  knowledge  of  human 
experts  by  adding  intelligence  to  measurement  systems, 
the  need  for  clear  and  unambiguous  rules  for  sampling  the 
performance  of  interest  (and  nothing  else  that  might  con¬ 
found  the  data) ,  and  organization  of  output  to  character¬ 
ize  the  quality  of  performance  in  a  way  that  is  useful  to 
the  user. 

Despite  difficulties,  there  has  been  progress  on 
performance  measurement  issues  over  the  years.  Some  of 
the  methods  using  engineering  data  bases  for  system  design 
studies  and  instructional  system  development  procedures 
for  training  have  provided  a  basis  for  defining  what  to 
measure  for  many  tasks.  Studies  of  empirical  performance 
measurenient  have  developed  methods  for  finding  and 
weighting  measures  of  importance  in  non-real  time,  and 
there  have  been  several  successes  in  the  development  and 
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validation  of  operational  performance  measures  (Brictson 
et  al.f  1973?  Lees  et  al.,  1976;  Ciavarelli,  1980),  and 
training  performance  measures  (Brictson  and  Burger,  1976; 
Waag  and  Knoop,  1977;  Vreuls  and  Wooldridge,  1977,  1981; 
Wooldridge  et  al.,  1982).  Solutions  to  real-time  measure¬ 
ment  issues  have  been  found  as  well  (Hennessy  et  al., 

1979;  Obermayer  et  al.,  1982).  There  is  much  promise  in 
the  approach  of  integrating  artificial  intelligence  tech¬ 
niques  into  performance  measurement  systems  for  training; 
performance  measurement  systems  can  be  made  "smarter"  to 
function  in  real  time  with  some  of  the  capabilities  of  an 
expert  observer.  Progress  has  also  been  made  on  the  kind 
of  architectures  and  algorithms  that  are  needed  to  control 
real-time  measurement  systems. 

The  efforts  to  date  have  been  relatively  small  and 
isolated.  A  greater  amount  of  coordinated  research  and 
development  effort  would  facilitate  development  of  gen¬ 
erally  applicable  procedures  for  building  useful  perfor¬ 
mance  measurement  systems. 


Recommendations 

For  each  use  of  simulation  there  are  performance 
measurement  issues  that  are  common  across  applications 
and  other  issues  unique  to  each  application.  The  working 
group  recommends  specific  research  and  development  to 
advance  performance  measurement  knowledge  and  practice  in 
the  design  and  use  of  simulators: 

1.  Operational  Performance  Criteria.  Operational 
measures  and  criteria  of  overall  system  effectiveness  are 
needed  for  representative  tasks  and  operating  environments 
(Mixon,  1981).  Except  for  a  few  tasks,  such  as  weapons 
delivery,  quantitative  criteria  for  acceptable  and  unac¬ 
ceptable  job  performance  are  usually  unknown  except  in 
very  general  qualitative  ways.  Yet  operational  criteria 
are  essential  to  any  simulator  performance  measurement 
system.  Establishing  operational  criteria  is  one  of  the 
most  critical  needs  to  improve  performance  measurement; 
empirical  data  collection  efforts  are  needed  for  repre¬ 
sentative  missions  and  tasks  performed  by  experienced 
personnel  to  establish  these  criteria. 

2.  Operator  Goals  and  Strategies.  Analysis  is  needed 
of  the  hierarchy  of  goals  and  control  strategies  that 
operators  employ  in  the  performance  of  real-world  tasks. 
During  training,  limited  goals  and  simplified  strategies 
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are  assumed  in  the  solution  recommended  by  the  school, 
but  these  may  not  be  appropriate  as  the  basis  for  perfor¬ 
mance  measurement  of  experienced  operators.  This  issue 
becomes  important  as  simulators  are  used  to  provide 
recurrent  training  for  experienced  operators  or  readiness 
training  for  military  combat  teams.  Research  is  needed 
to  derive  measurement  algorithms  that  are  sensitive  to 
which  goal  an  operator  is  pursuing  at  any  given  moment 
and  the  particular  control  strategy  being  used;  Connelly 
(1977)  has  demonstrated  the  value  of  measurement  systems 
with  this  characteristic. 

3 .  Performance  Diagnosis.  The  development  of  measure¬ 
ment  for  performance  diagnosis  is  a  major  research  issue 
in  its  own  right.  Diagnosis  becomes  difficult  when 
many  minor  deviations  from  expected  performance  compound 
into  error.  Considerable  analysis  effort  would  be 
required  to  provide  a  measurement  system  that  can  recog¬ 
nize  patterns  of  deviations  over  time  and  derive  the 
probable  root  causes.  In  addition,  the  architecture  for 
such  a  system  is  likely  to  be  very  different  from  the 
basic  simulator  control  system.  The  key  issue  is  the 
trade-off  between  the  value  of  detailed  diagnostic 
information  and  the  cost  of  obtaining  it.  Research  is 
required  to  determine  the  kind  and  level  of  detail  of 
diagnostic  information  required  for  training  and  system 
design  applications  of  simulators  (Obermayer  et  al., 

1982) . 

4.  Measurement  of  Team  Performance.  Most  complex 
systems  are  composed  of  teams  of  individuals  who  have 
designated  functions.  Except  for  highly  procedural 
tasks,  the  contribution  of  each  person  to  a  team  effort 
often  is  difficult  to  define  and  measure  (Crowe  et  al., 
1981) .  Performance  measurement  of  teams  has  been  and 
continues  to  be  one  of  the  most  important  topics  requir¬ 
ing  research  to  improve  the  use  of  simulators  for  design 
evaluation  and  training  of  multiperson  crews. 

5.  Automated  Performance  Measurement.  There  are 
reasons  for  automating  performance  measurement.  Automated 
measurement  can  be  based  on  a  greater  number  of  factors 
than  is  possible  through  direct  observation  by  humans; 
precision  and  reliability  can  be  improved.  Data  can  be 
collected,  summarized,  and  analyzed  in  a  short  period  of 
time;  adaptive  feedback  and  automated  training  can  be 
implemented;  personnel  requirements  and  thus  cost  can  be 
reduced. 

A  related  problem  is  simply  keeping  track  of  the  tasks 
being  performed  when  more  than  the  net  outcome  of  a 


simulation  exercise  is  of  interest.  End  goals,  i.e., 
outcomes  in  terms  of  system  parameters  or  effects,  are 
relatively  easy  to  measure.  However,  if  information 
about  the  processes  by  which  the  goals  are  reached  is 
necessary  for  the  diagnosis  of  training  or  the  evaluation 
of  equipment  and  procedures,  then  performance  measurement 
becomes  much  more  complicated.  The  system  must  know  how 
to  recognize  the  start  and  the  end  of  a  task.  This  is 
not  a  trivial  issue  because  many  real-world  operator  and 
maintainer  tasks  do  not  start  or  end  with  easily  recog¬ 
nized  discrete  events,  such  as  alarms  sounding. 

Common  measures  of  operator  control  describe 
operator -system  regulator  behavior  during  steady  states 
(e.g.,  holding  a  vehicle  on  course  or  in  a  turn  or  holding 
a  power  plant  at  a  given  power  level)  ,  but  this  is  only 
one  part  of  the  continuous  control  task.  Performance 
during  transitions  from  one  steady  state  to  the  next  is 
just  as  important  as  performance  during  the  steady  states 
(Connelly,  1977 ;  Vreuls  and  Wooldridge,  1981)  but  fre¬ 
quently  is  not  measured  because  the  algorithms  may  be 
complex  and  there  is  little  guidance  in  the  literature. 
Research  is  needed  to  develop  useful  algorithms  that 
describe  operator  behavior  during  transitions. 

In  addition,  many  tasks  are  performed  in  parallel; 
some  are  executed  only  once  and  others  may  continue  for 
long  periods  of  time.  Human  instructors  or  observers 
usually  can  take  these  changes  into  account;  automated 
systems  must  be  programmed  to  handle  task  changes.  To  do 
so  demands  a  substantial  amount  of  built-in  knowledge  and 
processing  capability.  Research  is  needed  to  develop  and 
validate  algorithms  that  recognize  transitions  in  tasks 
as  well  as  to  capture  the  essential  elements  of  perfor¬ 
mance  for  concurrently  performed  and  irregularly 
sequenced  tasks. 

6.  Near -Real-Time  Measurement.  Research  is  required 
to  develop  guidelines  and  procedures  for  near-real-time 
performance  measurement  systems  and  to  develop  hardware 
and  software  architectures  necessary  to  implement  these 
systems.  Development  of  an  easy-to-use  interface  to 
allow  an  experimenter,  test  director,  or  instructor  to 
rapidly  redirect  the  scenario  or  performance  measurement 
system  is  a  constituent  problem  for  near-real-time  per¬ 
formance  measurement  that  has  not  been  addressed  in  any 
general  way. 

7.  Observational  Performance  Assessment.  Automated 
instrumentation  and  performance  measurement  systems  are 
not  possible  in  all  simulations  or  at  least  for  certain 
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tasks.  There  are  important  behavioral  characteristics 
that  can  be  perceived  by  observers  that  are  very  difficult 
to  assess  by  automated  systems,  such  as  hesitations, 
smoothness  of  control,  the  physical  position  and  movements 
of  operators  in  a  control  area,  such  as  a  ship's  bridge 
or  a  nuclear  power  plant  control  room,  and  subtle  cues 
that  indicate  an  operator  or  controller  is  preparing  for 
an  oncoming  event  or  considering  alternative  problem 
solutions.  In  these  situations,  observers  frequently  are 
required  to  capture  the  performance  of  interest  as 
objectively  as  possible. 

Research  and  guidelines  are  needed  on  how  best  to  use 
observational  methods  and  to  decide  what  measurement 
functions  should  be  allocated  among  machines  and  humans. 
The  value  of  human  observers  should  not  be  forgotten  in 
the  pursuit  ot  automated  measurement  methods  for 
simulators. 

8 .  Education  in  Simulator  Performance  Measurement. 
There  is  no  single  source  (or  even  a  coherent  body  of 
literature)  to  which  practitioners  can  turn  to  obtain 
useful  data  on  performance  measurement  methods  and 
practice  in  simulation;  some  information  is  contained  in 
reports  on  empirical  studies,  but  this  literature  is 
diffuse  and  much  of  it  is  restricted  to  highly  specific 
problems  (Mixon  and  Moroney,  1982).  A  very  worthy  effort 
would  be  to  collect  and  describe  performance  measurement 
methods  in  simulators  in  a  form  useful  for  educational 
purposes  as  well  as  a  guide  for  practitioners. 

In  summary,  the  extensive  use  of  simulators  creates 
both  needs  and  opportunities  for  improving  human  perfor¬ 
mance  measurement  and  provides  the  means  to  automate  much 
of  the  process  as  well.  However,  there  is  much  fundamen¬ 
tal  work  to  be  done.  We  must  make  advancements  in  what 
to  measure  as  well  as  how  to  measure.  Basic  research  is 
needed  on  general  problems  such  as  developing  criteria, 
measurement  when  operator  goals  and  strategies  change, 
performance  diagnosis,  team  performance,  automated  per¬ 
formance  measurement,  particularly  on  a  near-real-time 
basis,  and  observational  assessment  of  performance. 
Education  in  performance  measurement  methods  in  simulation 
is  also  needed. 


Benefits 

Good  performance  measurement  information  is  the 
foundation  for  improving  quality,  timeliness,  and 


76 


cost-effectiveness  of  system  design,  training  system 
development,  certification  testing,  and  behavioral 
research.  The  potential  return  on  investment  in  perfor¬ 
mance  measurement  systems  is  vividly  demonstrated  by  tne 
results  of  a  study  that  developed  a  set  of  empirically 
derived  measures  to  automatically  control  advancement  of 
instrument  flight  lessons  in  a  training  simulator  (Vreuls 
et  al.,  1976).  The  empirically  derived  set,  when  used  in 
place  of  the  original  measures,  developed  analytically 
from  judgmental  criteria,  resulted  in  a  40  percent  reduc¬ 
tion  in  training  time  to  achieve  the  same  standard  of 
performance. 


TRAINING  SIMULATOR  INSTRUCTOR  GUIDE 

It  is  generally  recognized  that  the  validity  of  any 
simulator  training  program  is  dependent  on  the  appropriate 
use  of  the  simulator.  The  training  program  must  be 
designed  to  attain  specified  performance  objectives,  and 
the  instructor  must  be  trained  to  employ  the  device's 
instructional  features  in  a  manner  responsive  to  attain¬ 
ment  of  those  performance  objectives. 

The  instructor  is  a  key  element  in  the  conduct  of 
simulator  training.  Often,  his  or  her  qualification  for 
the  development  of  the  training  program  to  be  used  and 
for  the  conduct  of  that  training  consists  primarily  of 
skill  at  operation  of  the  system  simulated.  Such  an 
individual  may  have  no  training  with  respect  to  human 
learning  mechanisms  or  instructional  processes  and  tech¬ 
niques  that  can  facilitate  learning.  In  addition,  the 
instructor  seldom  has  information  in  a  usable  form  about 
the  various  instructional  features  of  the  simulator — 
features  that  make  the  device  a  training  tool  rather  than 
simply  a  less  than  perfect  rendition  of  the  operational 
system. 

Logistically  and  economically  it  is  not  always  possible 
to  provide  training  of  sufficient  intensity  and  duration 
to  all  simulator  instructors  to  make  them  fully  qualified 
to  perform  their  job  tasks.  Even  when  such  training  is 
feasible  for  a  small  group  of  instructors,  instructional 
resource  material  often  is  unavailable  or  is  not  in  a 
readily  usable  and  understandable  form.  The  absence  of 
appropriate  instructional  material  for  use  by  simulator 
instructors  during  their  training  and  as  job  aids  during 
their  subsequent  development  and  conduct  of  training 
programs  is  a  significant  deterrent  to  more  effective 
simulator  training. 
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Recommendations 

We  recommend  for  development  two  types  of 
instructional  material  and  job  aids  for  simulations 

1.  General  Concepts  of  Training.  This  material  should 
describe  the  human  as  a  learner  and  techniques  for  aiding 
him  or  her  in  the  acquisition  of  the  required  skills. 
Topics  to  be  addressed  should  include  guidance,  feedback, 
generalization,  discrimination,  learning  set,  cue  develop¬ 
ment,  the  difference  between  stimuli  and  cues,  transfer, 
and  other  concepts  basic  to  learning  (Spears,  1983).  An 
example  of  a  general  manual  that  addresses  many  of  the 
required  topics  is  the  Flight  Simulator  Utilization 
Handbook  (U.S.  Air  Force,  1953).  This  document  could 
readily  serve  as  the  model  for  development  of  a  modern, 
generic  guidebook  for  training  simulation  utilization. 

2.  Specific  Simulator  Features.  This  material  should 
prepare  the  instructor  for  the  conduct  of  training  in  a 
specific  simulator.  The  contents  should  describe  the 
simulator's  various  instructional  features  and  provide 
specific  guidance  for  their  use  during  the  process  of 
training  in  the  simulator.  The  material  should  serve  as 
a  job  aid  during  the  use  of  the  simulator  as  well  as  a 
text  during  the  instructor's  own  training  in  the  methods 
and  techniques  of  simulator  instruction.  An  example  of 
an  instructor  handbook  that  addresses  the  required  topics 
is  the  Instructor's  Guide  for  the  U.S,  Army's  UH-1  Flight 
Simulator  (Seville  Research  Corporation,  1979). 


Benefits 

The  benefits  of  better  trained  simulator  instructors 
will  be  reflected  in  the  qualifications  of  the  personnel 
they  train,  the  increased  efficiency  of  the  training 
process,  and,  in  all  likelihood,  in  the  increased  morale 
of  the  instructors  themselves. 


COGNITIVE  SKILLS 

The  role  of  the  operator  in  modern  systems  is  changing 
in  the  direction  of  supervisory  control  functions  and 
away  from  predominantly  psychomotor  tasks  (National 
Research  Council,  1983).  The  operator  in  the  past  served 
as  a  flexible  adaptive  link  in  sensing  and  control  systems 
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for  direct  operation  of  a  system.  His  functions  have 
become  more  managerial/  consisting  of  monitoring  system 
performance  for  conformity  to  plan,  assessing  situations, 
choosing  among  alternatives  in  accordance  with  pre- 
established  objectives,  and  evaluating  events  as  they 
occur.  These  cognitive  skills  are  poorly  understood  and 
not  well  conceptualized.  New  thinking  in  the  work  of 
Sternberg  (1977)  and  Hunt  (1983),  for  example,  and  on  the 
nature  of  expertise  in  the  area  of  artificial  intelligence 
are  only  beginning  to  address  this  research  area. 

The  tasks  of  the  human  as  a  direct  control  link  were 
primarily  simple  procedures  with  minimal  if  any  contin¬ 
gencies  or  branching.  The  operator  had  to  improvise  and 
make  decisions  if  nonstandard  or  emergency  conditions 
occurred?  however,  these  conditions  received  little 
consideration  during  system  design  or  formal  training. 

The  cognitive  functions  and  skills  of  the  operator's 
supervisory  role  are  semistructured  activities  (Keen  and 
Morton,  1978)  in  emergent  situations  (Boguslaw  and  Porter, 
1962)  for  which  the  sequences  of  specific  actions  are 
contingent  on  events  that  emerge  as  the  operational 
scenario  unfolds.  These  semistructured  tasks  cannot  be 
described  as  fixed,  invariant,  deterministic  sequences; 
rather,  they  must  be  represented  as  a  more  complex  tree 
structure  if  they  can  be  anticipated  or  preprogrammed  at 
all.  Operators  cannot  be  adequately  trained  on  these 
tasks  by  drill  on  a  fixed  scenario,  and  it  is  typically 
not  feasible  to  provide  training  on  an  adequate  sample  of 
unplanned  conditions.  Therefore,  research  is  needed  to 
determine  the  effectiveness  of  training  operators  on 
component  skills,  knowledge,  models,  and  rules  that  they 
use  in  real  time  to  analyze  specific  emergent  conditions 
and  fashion  procedures  and  actions  to  counter  threats  and 
maintain  progress  toward  mission  objectives. 

Advances  in  automation  and  electronics  have  made  it 
feasible  to  automate  fixed  procedural  tasks,  and  they  are 
now  beginning  to  enter  the  domain  of  these  semistructured, 
cognitive  functions.  Paradoxically,  automation  of  these 
tasks  tends  to  increase  the  interdependence  between  user 
and  machine  and  to  increase  the  complexity  of  the  oper¬ 
ator's  workload.  New  system  interface  concepts  are 
needed,  especially  in  high-technology  weapon  systems,  if 
they  are  to  provide  adequate  support  for  the  diversity  of 
information  processing,  speed,  and  accuracy  required  for 
operation  of  these  systems. 

Future  systems  must  be  designed  to  reduce  the  oper¬ 
ator's  workload  and  support  these  cognitive  activities. 
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They  must  provide  for  the  integration  of  data  from 
multiple  sensor  sources  into  situational  information, 
minimize  the  operator's  time  and  effort  devoted  to 
unaided  mental  processing  support  or  aid  tasks  requiring 
continuous  close  attention  in  a  multitask  environment, 
and  free  the  operator  of  time-consuming,  repetitive, 
algorithmic  operations  that  are  subordinate  elements  of 
cognitive  tasks. 

The  design  of  the  user-system  interface  to  be  effec¬ 
tive  and  efficient  must  also  be  based  on  and  incorporate 
users'  mental  models  of  the  systems  and  procedures  with 
which  they  are  working  (Stevens  and  Collins,  1977,  1980; 
Gentner  and  Stevens,  1983).  This  is  a  new  area  of  con¬ 
ceptualization  and  research,  which  is  essential  to 
maximizing  the  compatibility  between  system  and  user. 

The  critical  deficiency  for  simulation  in  future 
systems  is  the  lack  of  knowledge  about  how  and  what  to 
simulate  in  order  to  elicit,  observe,  and  train  cognitive 
skills.  An  individual  practitioner  of  the  simulation 
arts  might  have  good  pragmatic  insights  into  these 
processes,  but  they  are  not  known  in  the  sense  of  well- 
structured,  articulated,  and  shared  bodies  of  principles, 
rules,  and  supporting  data  that  are  necessary  for  a 
technology. 

The  emphasis  in  simulation  during  the  last  three 
decades  of  its  growth  and  acceptance  has  been  psychomotor 
skills,  especially  in  simulation  of  vehicular  operation 
and  control.  The  emphasis  in  battle  simulation,  however, 
has  been  largely  in  generating  a  realistic  scenario  of 
events,  updating  force  movement,  and  calculating  the 
outcomes  of  engagements.  A  review  of  force-on-force 
combat  models  typically  employed  by  the  Army  analysis 
community  revealed  that  behavioral  processes  and  human 
factors  are  considered  only  implicitly,  even  though  it  is 
well  recognized  that  human  participation  and  influence 
pervade  land  and  air  combat  (Miller  and  Bonder,  1982). 

The  fundamental  issues  for  simulator  training  of  cog¬ 
nitive  skills  can  be  broken  down  into:  (1)  identifying 
the  conditions  that  are  necessary  or  sufficient  for 
eliciting  cognitive  behaviors,  (2)  assessing  the  quality 
of  performance  and  level  of  mastery  in  cognitive  skills, 
(3)  identifying  or  building  a  taxonomy  and  operational 
definitions  of  cognitive  activities  and  skills  that  are 
adequate  for  describing  performance  of  tasks  with  cog¬ 
nitive  loadings,  and  (4)  developing  task  analytic 
techniques  for  the  decomposition  of  complex,  multitask. 
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time-sharing  operational  activities  into  their  constituent 
cognitive  as  well  as  psychomotor  components. 


Recomme  nda  t i on  s 


Research  on  cognitive  skills  should  be  initiated  to 
provide  the  fundamental  knowledge  leading  to  design  of 
adequate  user-system  interfaces  for  system  development, 
evaluation,  and  training.  This  research  should  include 
the  following  areas  as  a  minimum. 


1 .  Identify  the  Qualitatively  Different  Cognitive 
Functions  Performed  in  Modern  Systems.  Description  of 
the  functions  should  include  specific  actions  taken, 
input  knowledge,  and  output  of  each  action,  contingencies, 
and  boundary  conditions  that  affect  the  actions  taken  and 
the  user-machine  interaction  and  dependency  involved  in 
selecting  exemplary  systems,  e.g.,  flight  simulators  and 
nuclear  power  plant  simulators. 

2.  Develop  Operational  Definitions  and  Measures  of 
Performance  of  Cognitive  Skills.  These  measures  must 
permit  the  evaluation  of  capability  on  basic  cognitive 
abilities  as  well  as  provide  an  assessment  of  their 
effect  on  performance  of  operational  tasks. 

3.  Identify  the  Conditions  of  Simulation  That  Are 
Necessary  to  Generate  Representative  Cognitive  Activities. 
A  technology  adequate  for  effective  use  of  simulation  in 
system  development,  evaluation,  and  training  requires 
that  the  cognitive  skills  and  the  conditions  necessary 
for  their  performance  must  be  identified  in  both  generic 
form  and  in  the  domain-specific  context  of  the  system 
application.  The  skills  must  be  formulated  in  terms  of 
the  scenarios  and  scripts  typical  of  the  application 
domain. 


4 .  Develop  Cost-Effective  Methods  for  Training  Cog¬ 
nitive  Skills  Using  Simulation.  Both  new  methods  and 
content  will  be  involved  in  training  appropriate  to 
cognitive  skills.  Training  by  drill  and  memorization  of 
fixed  procedures,  which  applies  to  psychomotor  skills, 
does  provide  adequate  training  of  cognitive  skills.  The 
new  emphasis  must  consist  of  learning  basic  perceptual 
and  response  components  that  are  combined  flexibly  and 
adaptively  in  real  time  in  complex,  rapidly  changing, 
nondeterministic  situations. 

5 .  Develop  Concepts  and  Models  Describing  the  Nature 
and  Operation  of  the  User's  Mental  Model  in  the  User- 
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System  Interface.  The  user's  perception,  knowledge,  and 
concepts  of  the  equipment's  operation  and  the  job  that  he 
or  she  and  the  equipment  are  doing  are  the  stuff  of  the 
user's  mental  models;  they  are  critical  mediators  in  the 
user's  cognitive  processes.  The  models  are  functionally 
oriented  in  terms  of  the  job  activities  that  the  system 
performs  rather  than  the  technologies  of  hardware  and 
software  of  the  internal  mechanics  of  devices.  The  mental 
model  can  be  regarded  as  an  extension  of  the  concept  of 
"population  stereotypes"  for  control-display  relationships 
in  psychomotor  skills. 


Benefits 

This  research  is  basic  to  providing  operationally  ade¬ 
quate  interfaces  in  high-technology  systems  and  the  effec¬ 
tive  use  of  automation  in  future  system  concepts  and 
designs  as  well  as  methods  for  training  and  assessing 
cognitive  skills.  Knowledge  of  cognitive  skills  and 
their  underlying  behavioral  processes  should  also  enable 
development  of  adequate  operator  aiding  to  reduce  com¬ 
plexity  and  difficulty,  reduce  ability  aquirements,  and 
increase  effectiveness  in  the  use  of  simulator  training 
time. 


TRAINING  FOR  FAILURES  AND  OUT-OF-TOLERANCE  CONDITIONS 

One  of  the  earliest  uses  of  simulation  was  for  emer¬ 
gency  training.  Commercial  airline  training  focused  on 
instrument  training  and  accident  prevention,  while  mili¬ 
tary  aviation  emphasized  accident  prevention  plus  comple¬ 
tion  of  missions  with  failed  or  combat  damaged  systems. 

The  characteristics  of  emergencies  are  such  that  simu¬ 
lators  are  essential  for  training  and  proficiency  assess¬ 
ment  for  all  but  the  simplest  of  conditions.  Many  simu¬ 
lated  emergencies  are  too  dangerous  to  practice  in  the 
real  situation,  and  the  full  scope  of  symptoms  and  conse¬ 
quences  of  the  failure  and  out-of-tolerance  conditions 
needed  for  training  cannot  be  reproduced  in  operational 
equipment  (Jones,  1979). 

In  some  operational  cases  the  consequence  of  some 
inappropriate  actions  can  be  catastrophic,  while  others 
can  seriously  degrade  the  operational  effectiveness  of  a 
system  or  reduce  its  revenue-producing  potential.  Defici¬ 
encies  in  provisions  for  such  training  in  simulators  were 
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identified  years  ago  (Jones  and  Garrett,  1957),  but  many 
still  persist. 

Future  simulator  design  and  use  must  deal  with  the 
major  changes  occurring  in  advanced  systems.  For  example, 
newer  military  and  commercial  aircraft  will  have  program¬ 
mable  multifunction  controls  and  displays  driven  by 
computer-multiplex  bus  architecture  that  radically  change 
the  nature  of  subsystems  such  as  fuel  management  and 
weapons  and  flight  control.  They  present  a  different  set 
of  training  issues  from  the  past  in  that  there  may  be 
thousands  of  options,  many  automatically  configured,  but 
whose  operation  must  be  monitored  and  dealt  with  appro¬ 
priately  when  failures  or  out-of-tolerance  conditions 
occur . 

There  are  two  aspects  to  the  problem:  (1)  the  nature 
of  the  failure  or  out-of-tolerance  conditions  must  be 
identified  in  operator-oriented  terms  such  as  cues  and 
their  patterns,  decision  processes,  and  consequences  of 
action  and  (2)  the  explicit  nature  of  the  decision-making 
process  must  be  identified  and  reflected  in  the  design  of 
the  simulator  or  in  the  related  instructional  process. 


Recommendations 

A  two-pronged  research  approach  is  recommended:  (1) 
techniques  must  be  developed  for  identifying  the  charac¬ 
teristics  of  failures  and  out-of-tolerance  conditions 
during  system  acquisition,  including  multiple  failures, 
their  interactions  and  the  consequences  for  the  operators' 
related  decision-making  processes  and  (2)  principles  and 
practices  for  training  in  decision-making  (problem¬ 
solving)  must  be  developed.  Studies  on  these  problems 
need  to  be  developed  with  particular  emphasis  on  uncer¬ 
tainty  and  unanticipated  failures  for  the  type  of 
conditions  anticipated  in  future  systems. 

Such  a  research  program  would  consist  of  fundamental 
research  that  addresses  complex  decision  processes  and 
how  they  are  best  represented  and  taught  and  applied 
research  that  takes  an  integrative  approach  to  the  entire 
problem  of  dealing  with  failures  and  out-of-tolerance 
conditions.  Special  emphasis  should  be  placed  on  the 
decision-making  process  that  will  be  required  in  the 
"layered"  computer-driven  tasks  in  advanced  systems. 

The  integrative  approach  should  address  the  following 
issues : 
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1.  How  can  potential  failure  situations  be  best 
identified?  This  includes  the  range  of  failures,  how 
they  are  manifest  in  terms  of  cues  and  cue  patterns,  the 
impact  of  human  error  by  individuals  and  teams,  the 
consequence  of  the  failure,  and  the  role  of  engineering 
analyses  and  operating  experience  in  identifying  these 
situations. 

2.  How  can  means  for  creating  appropriate  failure 
situations  be  incorporated  into  simulators?  Designing 
for  this  purpose  must  take  into  account  the  necessary 
complexity  and  interactions,  implications  for  instructor 
station  design  and  aiding  techniques,  and  providing 
procedures  for  timely  updating  of  a  device  to  reflect 
operating  experience. 

3.  How  is  training  to  handle  complex  failures  best 
accomplished?  This  requires  research  on  improved  training 
techniques  drawing  on  fundamental  research  on  decision¬ 
making  and  judgment  training,  particularly  to  respond  to 
unanticipated  events,  the  important  issue  of  retention  of 
decision  skills,  and  its  implication  for  the  amount  and 
frequency  of  training.  The  value  of  multiple  complemen¬ 
tary  techniques  to  accomplish  this  training,  such  as 
computer-assisted  instruction  and  classroom  training, 
should  be  investigated. 

4.  How  is  the  capability  to  cope  with  failures 
assessed?  This  is  related  to  measures  of  individual  and 
team  proficiency  in  terms  of  speed,  accuracy,  and  opera¬ 
tional  consequences.  This  is  an  especially  difficult 
issue  when  more  than  one  pattern  of  behavior  is  appro¬ 
priate  to  solve  a  problem  and  the  training  is  directed  at 
classes  of  problems  rather  than  particular  problems. 

Finally,  documentation  of  the  existing  research 
findings  and  those  from  the  research  recommended  above  is 
needed  to  provide  design  guides  for  use  by  those  who 
develop  simulators  and  training  guides  for  those  who  use 
simulators. 


Benefits 

Appropriate  simulator -based  training  in  dealing  with 
failures,  particularly  those  that  are  unanticipated, 
should  result  in  reductions  in  accident  rates,  equipment 
damage,  aborted  missions,  equipment  shutdowns,  and 
improvements  in  rates  of  missions  successfully  completed 
with  degraded  equipment. 
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FUTURE  TRENDS  IN  SYSTEMS  AND  SIMULATION 


Forecasting  is  a  precarious  business.  However ,  we 
attempted  to  extrapolate  some  trends  that  are  apparent 
for  new  systems,  the  impact  of  these  on  the  role  of 
humans,  and  their  implications  for  simulation. 

Ne  expect  change  to  be  evolutionary,  in  that  concepts 
and  technology  that  are  embryonic  now  will  be  exploited 
in  the  future.  If  one  looks  back  on  the  progress  in 
simulation  during  the  past  three  decades,  there  is  little 
that  is  radically  new,  with  the  exception  of  digital 
computer  technology,  which  permitted  the  realization  and 
exploitation  of  earlier  ideas.  It  might  be  noted  that 
many  systems  such  as  aircraft,  nuclear  power  plants,  com¬ 
mand  and  control  networks  and  weapons  systems  that  are 
entering  the  inventory  and  being  simulated  today  may 
still  be  in  use  20  to  25  years  from  now. 


SYSTEMS 

It  is  not  possible  to  project  the  full  range  of  new 
systems  and  vehicles  that  might  be  simulated.  We  do  know 
that  the  trend  has  been  to  apply  simulation  more  broadly. 
We  expect  that  newer  systems  will  have  characteristics 
such  as  greater  complexity  and  be  more  highly  automated 
and  integrated  and  involve  fewer  operators  or  crew  members 
(National  Research  Council,  1982).  Systems  will  continue 
to  expand  the  use  of  more  capable  computers,  and  therefore 
be  more  software-driven  and  have  many  forms  of  artificial 
intelligence  implemented. 

Controls  and  instruments  will  be  multifunctional,  with 
fewer  special  purpose  or  dedicated  instruments.  Informa¬ 
tion  will  undergo  more  processing  and  be  integrated  in 
pictorial  formats.  Control  devices  will  affect  processes 
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at  a  higher  level  and  will  be  more  for  the  purpose  of 
conveying  commands  and  intentions  than  continuous  psycho¬ 
motor  tracking  or  adjustment  of  low-level  processes. 
Functions  such  as  navigation,  systems  regulation,  and 
target  acquisition  will  be  automated  to  a  larger  degree. 

Systems  will  expand  to  include  many  levels  and  widely 
dispersed  entities  by  virtue  of  multiple  voice  and  data 
communication  links.  Information  acquisition,  analysis, 
synthesis,  and  distribution  will  occur  with  little  or  no 
direct  involvement  of  humans  except  as  receivers  or  when 
occasional  direct  conversation  is  necessary. 

Maintenance  will  also  become  more  automated,  with  self¬ 
diagnostic  and  fault  identification  functions  built  into 
systems.  Systems  will  be  fault  tolerant,  working  around 
failed  components,  and  degrading  gracefully  rather  than 
precipitously  and  catastrophically. 


THE  ROLE  OF  HUMANS  IN  FUTURE  SYSTEMS 

The  role  of  humans  follows  from  the  system  design, 
which  is  a  function  of  operational  requirements,  avail¬ 
able  technology,  the  cost  of  the  human  component,  and 
whether  the  system  is  directly  or  remotely  controlled. 
Maintenance  functions  for  humans  are  driven  by  logistical 
concepts  as  well  as  system  and  support  equipment  tech¬ 
nology. 

The  shift  in  the  function  of  the  human  from  a  low-level 
system  operator  and  continuous  controller  to  a  system 
supervisor,  i.e.,  monitor  and  decision  maker,  will  con¬ 
tinue.  Many  of  these  functions  might  be  done  remotely 
through  some  sort  of  human  interface  system,  several 
levels  above  autonomous  self-controlling  task  processes. 
For  example,  this  trend  is  apparent  in  aircraft  such  as 
the  F-18  and  its  interface  with  the  Joint  Tactical 
Information  Distribution  System  (JTIDS) .  Some  of  the 
specific  implications  for  the  future  role  of  humans  in 
systems  are: 

*  Psychomotor  tasks  related  to  vehicle/weapons 
control  will  be  deemphasized.  Tasks  requiring 
cognitive  skills  and  supervisory  control  will  be 
dominant. 

*  More  system  options  will  increase  the  variety  and 
complexity  of  tasks. 

*  Greater  emphasis  will  be  placed  on  information 
assimilation  and  decision  making. 
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Fewer  individuals  will  be  required  and  their 
placement  in  the  total  system  may  be  different. 
New  technologies  such  as  voice  recognition  and 
pictorial  displays  will  add  new  types  of  tasks. 
Higher  levels  of  operator/controller  proficiency 
will  be  required. 

Operators/controllers  will  require  a  broader 
system  perspective  regarding  how  they  contribute 
to  the  total  operational  picture. 

Workload  will  remain  high  despite  automation 
because  of  the  addition  of  new  functions. 


SIMULATORS 

Probably  the  most  important  potential  for  design  and 
training  simulation  is  its  use  in  the  integration  of 
complex  system  elements.  For  example,  the  design  of  any 
advanced  military  aircraft  must  consider  not  only  the 
vehicle,  but  its  interface  with  other  aircraft,  the 
system,  the  maintenance  logistics  system,  and  the  tactical 
environment  in  which  it  must  operate.  Hie  latter  includes 
not  only  the  physical  features  of  the  terrain,  but  also 
important  military  elements  such  as  missiles,  radar,  and 
countermeasures.  A  federated  simulation  system  augmented 
with  models  can  duplicate  these  elements  as  a  part  of  the 
design,  development,  and  test  and  evaluation  process. 

Not  only  can  the  human-machine  elements  be  considered, 
but  software  and  prototype  equipment  used  in  the  design 
simulator  can  be  transferred  directly  to  the  training 
s imulator . 

It  is  expected  that  the  use  of  simulators  will  increase 
as  a  function  of  the  economics  of  training  and  system 
design  as  well  as  the  need  to  duplicate  situations  that 
are  too  risky  or  complex  in  actual  practice.  However, 
there  may  well  be  a  proliferation  of  low-cost,  portable 
devices  made  possible  by  new  low-cost  computer  technology. 
At  the  same  time,  high-cost,  integrated  simulation  systems 
may  be  developed  in  which  many  devices  are  linked  for 
interactive  use  in  real  time.  Or,  if  there  is  an  increase 
in  embedded  simulation,  existing  communication  links  can 
be  used  for  training  as  well  as  operational  purposes. 

It  is  expected  that  newer  simulators  will  be  more 
compact  and  reliable  and  that  deliberate  design  decisions 
will  be  made  to  readily  permit  the  modifications  necessary 
to  keep  the  simulator  current  with  the  device  it  repre¬ 
sents.  It  might  be  noted  that  design  simulators  have 
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provisions  that  permit  the  complete  change  of  configura¬ 
tions  in  several  hours.  These  modification  concepts  have 
been  applied  to  training  simulator  design  to  a  much  more 
limited  extent. 

Some  of  technologies  that  will  have  greater  emphasis 
in  new  devices  are: 

*  Operator  and  Instructor  Station  Design 

-  Artificial  intelligence 

-  Voice  synthesis  and  recognition 

-  Alphanumeric,  graphic,  and  combination  displays 

*  Computer  Generated  Imagery 

-  Detailed  duplication  of  natural  environments 

-  Artificial  and  augmenting  information  added  to 
scenes 

Although  the  trends  in  simulation  given  above  will  charac¬ 
terize  all  uses  of  simulation,  there  may  be  special 
considerations  for  specific  uses. 


Design  Use 

Design  simulation  for  human  factors  purposes  has  only 
occasionally  used  the  more  complex  integrated  simulators 
that  include  high-fidelity  environmental  and  situational 
conditions.  Such  uses  are  not  only  valuable  for  initial 
design,  but  can  also  extend  through  test  and  evaluation. 

A  current  example  is  the  extensive  testing  of  a  new 
missile  in  a  simulated  tactical  environment  with  projected 
threat  aircraft.  We  expect  to  see  more  examples  of  this 
use  for  "fly  offs,"  particularly  when  the  C3  elements 
of  the  system  are  included.  These  uses  require  objective 
individual,  system,  and  mission  performance  measurements. 
We  can  expect  to  see  improved  measurement  capability  with 
data  available  to  the  designer  and  evaluator  on  a  near- 
real-time  basis.  These  measures  can  also  serve  as  a 
source  of  empirically  derived  training  requirements  as 
by-products  of  the  use  of  simulators  for  design. 


Training  Use 

The  trend  of  substitution  of  simulator  time  for 
operational  practice  and  experience  time  will  accelerate 
both  for  economic  reasons  and  because  many  operational 
tasks  cannot  be  duplicated  in  the  real  world.  The  com- 
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mercial  aiclines  and  NASA  have  been  leaders  in  this  area; 
the  Apollo  and  Shuttle  programs  are  prime  examples.  How¬ 
ever,  the  military  services  prefer  to  view  simulators  as 
a  means  to  augment  the  value  of  actual  flight  rather  than 
as  a  substitute  for  aircraft  training. 

There  will  be  an  increased  emphasis  on  training  cogni¬ 
tive  skills  to  reflect  the  changing  role  of  the  operators. 
The  instructional  functions  will  introduce  artificial 
intelligence  concepts  in  areas  such  as  performance  assess¬ 
ment,  feedback,  and  adaptive  training  to  help  improve  the 
quality  of  instruction  and  to  reduce  the  demands  on 
instructors.  Computer -managed  instruction  (CMI)  may  be 
incorporated  either  into  the  basic  simulator  or  in  sup¬ 
plementary  devices  as  a  way  of  planning  and  implementing 
the  overall  training  process  more  effectively.  Part-task 
devices  will  be  used  when  appropriate  and  greater  use 
will  be  made  of  Computer  Aided  Instruction  (CAI)  with 
improved  computer  graphics  for  lead-in  or  special  instruc¬ 
tion  not  appropriate  in  the  larger,  more  expensive  simu¬ 
lators.  Simulators  will  incorporate  better  measurement 
systems  to  measure  proficiency  in  important  operational 
tasks.  The  trend  toward  maintenance  simulation  will 
continue,  reflecting  to  some  extent  the  increased  use  of 
automatic  test  equipment.  C3  training  will  reemerge  as 
an  important  area,  with  embedded  training  features  an 
integral  part  of  many  computer-based  systems. 


Research  Use 

With  some  few  exceptions,  high-fidelity  simulators 
duplicating  operational  tasks  have  not  been  used  for 
behavioral  science  research.  However,  advancements  in 
the  technology  necessary  to  support  human  factors  engi¬ 
neering  is  dependent  on  data  derived  from  operational 
tasks  that  consider  the  complex  and  interactive  effects 
of  environmental  conditions  and  events.  The  extensive 
use  of  human  performance  models  in  conjunction  with 
simulation  involving  human  participants  has  untapped 
potential  that  we  hope  to  see  realized  in  the  future. 


Licensing  and  Certification  Use 

There  is  little  doubt  that  a  principal  future  use  of 
simulators  will  be  for  licensing  and  certification.  At 
present,  the  principal  application  of  simulators  for  this 


purpose  is  for  commercial  airline  pilots.  For  more  than 
10  years,  the  Coast  Guard  has  used  simulation  in  certi¬ 
fication  of  pilots  for  instrument  flight  proficiency. 

The  Nuclear  Regulatory  Commission  is  studying  requirements 
for  simulator-based  test  and  licensing  of  nuclear  power 
plant  operators  (Ranken  et  al.,  1984).  Preliminary  steps 
are  also  being  taken  for  test  and  licensing  of  ship 
masters  in  ship  handling  simulators  (Hammell  et  al., 

1980) .  He  foresee  the  use  of  many  military  and  civilian 
simulators  for  certification  and  licensing  of  many  types 
of  systems  operators  and  maintainers. 
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CONCLUSIONS  AND  GENERAL  RECOMMENDATIONS 


There  has  been  a  rapid  growth  of  simulation  during  the 
past  quarter  of  a  century  in  both  traditional  and  new 
areas.  Initial  uses  focused  on  training,  but  research 
and  design  applications  have  become  increasingly  impor¬ 
tant.  This  growth  is  due  to  the  considerable  economic 
advantage  and  risk  avoidance  inherent  in  the  use  of 
simulators  for  training  and  design.  Simulation  permits 
situations  to  be  reproduced  artificially  and  examined  in 
ways  that  are  not  practical,  feasible,  or  affordable  in 
the  real  world.  There  is  no  question  that  simulation  is 
being  used  effectively,  but,  with  certain  technical 
advances  and  more  emphasis  on  behavioral  factors,  it  has 
the  potential  of  making  even  greater  contributions. 

The  national  investment  in  the  procurement  and  opera¬ 
tion  of  simulators  probably  involves  several  billion 
dollars  a  year .  The  total  spent  on  simulator  research 
and  development  approaches  $150  million  per  year.  Most 
of  this  amount  supports  hardware  and  software  develop¬ 
ments  and  demonstrations;  a  small  percentage  is  devoted 
to  research  on  behavioral  factors  in  simulation.  Yet  the 
working  group  sees  behavioral  research  as  the  primary 
basis  for  improving  the  design  and  use  of  simulation  by 
DoD,  NASA,  their  supporting  academic  and  industrial  base, 
and  other  major  users  of  simulation  such  as  the  electric 
power  and  maritime  industries. 

Behavioral  issues  and  research  recommendations  have 
been  covered  in  a  number  of  reports  going  back  almost  30 
years.  We  did  not  treat  these  issues  in  detail — the 
authors  of  those  reports  made  the  points  well  enough — but 
simply  reaffirmed  that  these  issues  are  still  important 
and  the  recommended  research  still  deserves  attention. 

We  do  make  specific  research  recommendations  on  several 
topics  that  we  think  are  particularly  important.  Most  of 
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these  topics  have  either  been  neglected  altogether  or 
have  not  been  accorded  the  emphasis  they  deserve. 

Our  purpose  in  this  concluding  chapter  is  not  to 
repeat  or  consolidate  the  specific  research  conclusions 
and  recommendations  found  in  the  literature  cited  or 
discussed  in  other  sections  of  this  report.  Rather,  we 
take  a  broad  view  of  the  concerns  that  persist  in  a 
variety  of  simulator  types  and  applications.  From  this 
perspective  we  try  to  characterize  the  nature  of  the 
fundamental  problems  that  we  see  to  be  the  roots  from 
which  stem  the  many  specific  issues  in  simulation.  He 
present  eight  conclusions  that  we  perceive  to  be  the 
problems  that  must  be  overcome  to  bring  about  general 
improvement  in  the  use  and  design  of  simulators.  These 
conclusions  and  their  interrelationships  are  briefly 
described  in  the  next  few  paragraphs  since  it  is  important 
to  understand  these  relationships  to  appreciate  the 
significance  and  implications  of  the  conclusions  as 
problems  to  be  overcome. 

The  origin  of  many  of  the  current  problems  is  over¬ 
reliance  on  the  belief  that  the  degree  of  physical 
fidelity  of  a  simulator  is  the  principal  determinant  of 
its  capability  to  serve  its  intended  purpose.  At  the 
same  time,  the  importance  of  the  relationship  between  how 
a  simulator  is  used  and  its  effectiveness  is  not  fully 
appreciated.  As  a  consequence  of  this  imbalance  of 
emphasis,  simulators  are  often  not  as  cost-effective  as 
they  could  be  and  the  contributions  that  behavioral 
science  and  human  factors  engineering  can  make  to  the 
design  and  use  of  simulators  are  relatively  neglected. 

Many  serious  issues  relevant  to  stimulator  effectiveness 
have  persisted  because  of  the  failure  to  recognize  that 
solutions  to  problems  common  to  a  variety  of  simulator 
applications  must  come  from  a  better  understanding  of  the 
basic  behavioral  processes  involved  rather  than  engineer¬ 
ing  improvements  or  fixes. 

Although  simulators  could  be  improved  by  better  use  of 
existing  behavioral  and  human  engineering  principles,  the 
potential  contribution  of  these  fields  is  limited  by  the 
lack  of  adequate  performance  measurement  methods  and  well- 
developed  human  performance  models.  Both  are  impediments 
to  improving  the  effectiveness  and  efficiency  of  simu¬ 
lators  as  well  as  conducting  the  behavioral  research 
necessary  to  solve  many  persistent  simulator  problems. 
Objective  and  automated  performance  measurement  systems 
would  greatly  facilitate  the  derivation  of  detailed 
quantitative  performance  information  useful  for  design. 
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training,  and  research  purposes.  Moreover,  information 
in  this  form  is  necessary  for  the  further  development  of 
models  of  human  performance  that  have  a  variety  of  appli¬ 
cations  relevant  to  simulation.  For  example,  models  can 
replace  some  human  participants  in  system  evaluation 
studies  and  training  exercises.  More  fundamentally, 
quantitative  human  performance  models  provide  a  mechanism 
for  accumulating  and  refining  behavioral  knowledge  as 
well  as  conveying  this  information  to  system  designers 
(including  simulator  designers)  in  a  form  they  are 
accustomed  to  using. 

This  latter  use  of  models  may  bring  about  a  solution 
to  what  we  see  as  the  fundamental,  overriding  problem — the 
lack  of  an  integrated  science  and  technology  of  simula¬ 
tion.  The  contributions  of  engineering  and  behavioral 
science  to  achieving  the  goals  of  simulation  are  uncoor¬ 
dinated  at  best.  The  design  and  use  of  simulators  should 
be  based  on  an  integrated  multidisciplinary  approach  that 
draws  on  knowledge  of  engineering,  behavioral  science, 
and  computer  science  as  well  as  other  fields  that  bring 
into  balance  concerns  about  physical  correspondence 
between  real  systems  and  simulators  and  how  these  factors 
contribute  to  the  effectiveness  of  simulators. 

The  working  group  has  formulated  three  key  recommenda¬ 
tions  that  we  believe  are  essential  to  creating  the 
conditions  necessary  to  solve  the  fundamental  problems  in 
simulation  and  thereby  attack  the  roots  of  the  many 
specific  problems  that  we  and  others  have  identified. 


CONCLUSIONS 

1.  Physical  Correspondence  of  Simulation  is  Over¬ 
emphasized  for  Many  Purposes,  Especially  Training. 

A  persistent  question  for  simulation  has  been  how 
accurate  a  reproduction  of  operational  functions  is 
needed  to  achieve  the  intended  purpose.  Many  concerns 
about  physical  fidelity  or  correspondence  center  on 
emulating  system  dynamics,  visual  and  motion  cueing,  and 
representing  malfunction  or  degraded  mode  characteristics. 
In  the  past,  fidelity  tended  to  be  bounded  by  hardware 
capability.  Recent  increases  in  computer  and  display 
capabilities  have  made  possible  simulators  with  great 
complexity  and  realism  but  at  high  cost. 

Our  knowledge  regarding  the  need  for  physical  cor¬ 
respondence  is  limited  and  fragmented.  Determining 
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simulator  requirements  to  meet  behavioral  objectives  has 
been  largely  guesswork  using  analytical  frameworks 
augmented  by  some  research  results.  Most  evaluations  of 
simulators  are  global  in  nature  and  tell  us  little  about 
the  relative  contribution  of  specific  hardware  and  use 
variables.  One  result  of  this  lack  of  knowledge  has  been 
to  increase  fidelity  by  building  more  elaborate  equipment 
than  may  be  needed.  Because  effective  use  of  simulators 
depends  on  achieving  the  same  behavioral  outcomes  that 
occur  in  operational  situations,  the  concern  with  fidelity 
should  shift  from  what  is  technically  feasible  in  a  hard¬ 
ware  sense  toward  achieving  greater  effectiveness  and 
efficiency  in  terms  of  behavioral  objectives. 

2.  Simulators  Are  Often  Not  Used  Properly. 

The  utility  of  a  simulator  is  as  much  a  function  of 
its  method  of  use  as  of  its  physical  correspondence  to 
the  system  or  equipment  simulated.  While  good  utiliza¬ 
tion  techniques  can  compensate  for  reduced  physical 
fidelity,  a  simulator  that  is  a  high-quality  physical 
representation  of  a  system  can  be  relatively  ineffective 
if  it  is  not  used  properly.  In  many  cases  the  available 
capabilities  of  simulators  are  not  used  fully  or  expertly. 

Some  components  that  contribute  to  quality  of  use  are 
scenarios,  training  techniques,  operating  and  test  pro¬ 
cedures,  instructor  and  operator  knowledge  and  skills, 
proficiency  assessment  methods,  and  support  features  such 
as  properly  designed  control  consoles  and  instructor/test 
director  stations.  Design  of  these  components  is  largely 
based  on  front-end  analysis.  This  process  requires  con¬ 
siderable  specialized  effort  and,  for  cost  reasons  and 
lack  of  appreciation  of  its  ultimate  importance,  is  often 
neglected.  This  has  been  the  finding  of  many  past  studies 
and  has  been  highlighted  in  a  recent  Government  Accounting 
Office  report  (CJ.S.  Government  Accounting  Office,  1983)  . 

Physical  requirements  for  simulators  cannot  be  con¬ 
sidered  in  isolation  because  they  interact  with  the  way  a 
device  is  used.  We  need  to  know  more  about  the  nature  of 
these  use  factors  and  their  interactions  with  equipment 
factors  as  a  basis  for  simulator  design.  Greater  emphasis 
needs  to  be  placed  on  the  systematic  derivation  of  these 
use  factors  to  establish  requirements  that  are  more 
directly  relevant  to  the  processes  of  learning  and 
performance  elicitation. 
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3.  Simulation  Could  Be  More  Cost-Effective. 


As  a  consequence  of  the  overreliance  on  physical  cor¬ 
respondence  and  the  relative  neglect  of  use  factors, 
simulators  are  less  cost-effective  than  they  could  be. 

The  drive  to  technically  improve  the  realism  and  compre¬ 
hensiveness  of  simulators  has  major  cost  consequences. 
Simulators  could  be  more  cost-effective  if  greater 
emphasis  is  given  to  understanding  how  factors  such  as 
physical  realism  and  comprehensiveness,  methods  of  use, 
and  user  acceptance  influence  the 
attainment  of  behavioral  objectives. 

4 .  The  Role  of  Behavioral  Science  and  Human  Factors 
Engineering  Simulation  is  Neglected. 

Behavioral  and  physical  aspects  of  simulation  tend  to 
be  viewed  separately  and  independently.  Considerations 
such  as  perception,  learning,  retention  of  skills,  per¬ 
formance  validity,  performance  assessment,  and  human 
factors  engineering  are  seldom  regarded  as  considerations 
integral  to  the  development  of  simulator  hardware  and 
software  requirements.  Equipment  and  functions  are  more 
tangible  and  tractable  entities  than  behavioral  processes 
and  performance.  In  addition,  costs  can  be  more  easily 
and  directly  associated  with  hardware  products  than  with 
behavioral  outcomes. 

Simulator  procurement  specialists  and  users  of  simu¬ 
lators  who  generally  are  operational  system  experts  share 
an  understanding  of  hardware  technology;  but  usually 
neither  group  is  well  versed  in  training,  human  perfor¬ 
mance  measurement,  or  human  factors  engineering.  Con¬ 
sequently,  when  procurement  specialists  and  users  cooper¬ 
ate  to  establish  requirements  for  a  simulator,  hardware 
and  functional  correspondence  between  the  simulator  and 
the  target  system  tend  to  be  emphasized  and  behavioral 
considerations  tend  to  be  neglected. 

Even  when  the  procurement  and  user  personnel  are  sen¬ 
sitive  to  behavioral  considerations,  the  design,  procure¬ 
ment,  and  employment  of  simulators  are  separate  activities 
performed  with  little  coordination  and  integration.  Hie 
important  connections  among  the  purpose,  method  of  use, 
and  design  of  a  simulator  are  broken.  Yet  it  is  at  these 
connections  that  behavioral  science  and  human  factors 
engineering  can  have  a  positive  impact  on  simulator 
effectiveness . 
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5.  Many  Persistent  Simulation  Issues  are  Common  Across 
Types  and  Uses. 

An  examination  of  the  research  literature  and  reports 
of  science  advisory  groups  for  the  past  30  years  reveals 
that  most  problems  are  common  to  most  simulation  contexts, 
have  been  identified  repeatedly,  and  have  persisted.  Such 
fundamental  issues  as  cue  requirements,  learning  and  its 
transfer,  validity  of  performance,  control  of  simulation 
and  instructional  processes,  research  and  test  design 
strategies,  and  performance  measurement  occur  in  most 
simulation  contexts.  The  differences  in  issues  from  one 
type  of  application  to  another  are  superficial.  That  is, 
the  system  represented,  the  tasks  required,  and  the  pur¬ 
pose  (i.e.,  design,  training,  research,  or  certification) 
do  not  substantially  affect  what  fundamentally  needs  to 
be  known  or  the  approach  necessary  to  solve  the  problems. 

These  problems  persist  because  their  solutions  depend 
on  a  better  understanding  of  basic  behavioral  processes 
relevant  to  simulation.  The  necessary  knowledge  will 
come  largely  from  research  that  is  dedicated  to  the 
discovery  and  development  of  broad  principles.  This  sort 
of  research  requires  the  commitment  of  substantial 
resources  over  a  long  period  of  time.  To  date,  no  such 
commitment  has  been  made.  Most  behavioral  research  for 
simulation  has  consisted  of  short-term  projects  dedicated 
to  solving  what  is  perceived  to  be  specific  application 
problems.  This  kind  of  research  does  not  produce  results 
that  can  be  generalized  to  other  simulator  types  and  uses. 

6.  Our  Capability  to  Measure  Operator  Performance  is 
Limited. 

Measurement  is  the  keystone  for  design,  research,  and 
the  assessment  of  proficiency  in  simulation.  However, 
adequate  capabilities  do  not  exist  to  obtain  objective 
relevant  human  and  human-system  performance  measures. 
Although  this  deficiency  has  been  recognized  for  some 
time,  the  interdisciplinary  programs  necessary  to  solve 
the  problem  have  not  been  forthcoming.  The  lack  of 
capability  to  measure  performance,  particularly  on  a 
near-real-time  basis,  impedes  both  the  current  utility  of 
simulators  for  training,  systems  design,  and  licensing 
and  the  execution  of  the  fundamental  behavioral  research 
necessary  to  improve  future  simulator  design  and  use. 
Recently,  the  DoD  Defense  Science  Board  Summer  Study  on 
Training  and  Training  Technology  (1982)  highlighted  the 
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importance  of  the  performance  measurement  problem,  and 
the  secretary  of  defense  directed  the  services'  sec¬ 
retaries  to  support  efforts  .  .to  develop  performance 
measures  and  criteria  for  use  in  determining  performance 
levels  and  cost  effectiveness  of  alternative  method¬ 
ologies"  (U.S.  Department  of  Defense,  1983). 

7.  The  Use  of  Modeling  in  Simulation  is  Not  Well 
Developed. 

Human  performance  models  both  describe  and  predict 
behavior  under  the  circumstances  encompassed  in  the 
model.  Human  performance  modeling  and  human-in-the-loop 
simulations  can  have  important  complementary  relation¬ 
ships  for  research  and  system  design  purposes.  An  impor¬ 
tant  consequence  of  human  performance  models  is  that  they 
express  data  on  human  behavior  in  a  form  familiar  to  and 
usable  by  engineers  and  designers.  The  development  of 
these  models  thus  offers  the  potential  not  only  to  docu¬ 
ment  knowledge  of  behavior  in  a  rigorous  fashion  but  also 
to  provide  a  communication  medium  for  incorporating  this 
knowledge  into  the  simulator  design  process. 

Human  performance  models  have  many  applications  in  the 
use  of  simulators  as  well.  Modeling  can  be  done  early, 
be  relatively  economical,  consider  stressor  effects  on 
humans  that  are  hazardous,  and  effect  economies  in  simu¬ 
lation  through  early  sensitivity  analyses. 

Training  simulation  also  can  benefit  from  models.  An 
obvious  application  is  to  simulate  team  members  or  func¬ 
tions  not  actually  present  during  the  simulation.  Models 
might  be  used  in  developing  training  curricula  and  para¬ 
digms  of  performance  as  well  as  in  specifying  simulator 
design  requirements  regarding  dynamic  response  fidelity 
and  cue  fidelity.  Recent  developments  in  intelligent 
systems  and  knowledge  representation  will  also  permit 
modeling  of  the  instructor's  pedagogical  functions, 
student  characteristics,  and  the  learning  process  for  use 
in  instructional  strategies  and  performance  assessment. 

Relatively  limited  use  has  been  made  of  these  comple¬ 
mentary  relationships,  especially  for  training;  however, 
they  should  be  of  increasing  importance  in  the  future. 
Further  developments  needed  to  enhance  these  capabilities 
are  currently  being  addressed  by  the  working  group  on 
human  performance  modeling  of  the  Committee  on  Human 
Factors . 
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8.  The  Science  and  Technology  of  Simulation  Are  Not  Well 
Developed  or  Integrated. 

Perhaps  our  key  finding  is  that  there  is  not  a  well- 
developed  or  integrated  science  and  technology  of  simula¬ 
tion.  Integration  applies  not  only  within  pertinent 
disciplines  but  between  disciplines,  particularly  between 
the  engineering  and  behavioral  sciences.  Furthermore,  we 
are  aware  of  no  academic  institution  that  offers  formal 
integrated  engineering  and  behavioral  science  training 
specific  to  simulator  design,  development,  and  use. 

The  lack  of  incorporation  of  behavioral  science  and 
human  factors  engineering  into  the  technology  base  of 
simulation  is  especially  evident.  A  comprehensive  body 
of  behavioral  principles  and  methods  relevant  to  simula¬ 
tion  does  not  exist.  Integrative  assessments  of  the 
research  and  operating  experience  in  the  form  of  books 
and  other  documents  are  almost  completely  lacking  for 
important  topics  such  as  fidelity,  performance  measure¬ 
ment,  and  visual  and  other  cueing  systems.  A  result  is 
that  there  is  no  history  of  simulation  research  and 
utilization  technology  in  a  readily  accessible  form  that 
serves  as  a  guide  for  the  future.  Much  material  has  been 
lost  or  neglected  and,  as  a  result,  many  simulator 
studies  have  been  repeated  over  the  past  30  years. 

The  state  of  knowledge  today  can  best  be  characterized 
as  fragmented.  At  present  there  is  not  an  institutional 
base  of  significance  that  deals  with  the  spectrum  of 
fundamental  issues  and  application  in  simulation.  These 
points  were  emphasized  in  the  recent  review  of  DoD  lab¬ 
oratories,  which  states  that  the  fractioning  of  the  effort 
has  resulted  in  a  .  .  lack  of  focus  on  generic  re¬ 
search,  common  development,  maintaining  a  repository  of 
information  about  the  state-of-the-art,  and,  most  impor¬ 
tantly,  the  development  of  a  community  of  skilled  profes¬ 
sionals  in  the  simulation  discipline"  (Herman,  1982). 


GENERAL  RECOMMENDATIONS 

1.  Long-Range,  Comprehensive,  and  Forward-Looking 
Research  Plans  Should  Be  Developed  to  Address  Persistent 
and  Emerging  Simulation  Problems. 

Our  study  has  identified  the  fact  that  most  of  the 
design  and  utilization  issues  in  simulation  are  common  to 
the  various  types  and  applications  of  simulation.  Despite 
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this  fact,  many  agencies'  research  programs  tend  to  be 
limited  to  specific  applications  because  most  users  feel 
their  problems  are  unique.  They  have  not  recognized  that 
programs  of  fundamental  research  can  be  augmented  by 
situation-specific  applied  research.  The  result  has  been 
that  research  on  simulator  issues  during  the  past  three 
decades  has  tended  to  be  relatively  narrow  in  focus  with 
many  investigations  of  similar  problems  in  different 
contexts. 

Recent  progress  has  been  noted  in  better  integration 
and  planning  of  simulation  research  within  government 
agencies,  particularly  DoD  and  NASA.  However,  further 
improvements  are  needed  along  with  inclusion  of  univer¬ 
sities  and  the  industrial  base  with  its  independent 
research  and  development  (IRAD)  program.  Persistent 
problems  should  be  systematically  addressed  in  long- 
range,  comprehensive,  and  forward-looking  research  plans 
and  should  involve  all  segments  of  the  simulation  com¬ 
munity  concerned  with  advanced  hardware  technology,  the 
roles  of  humans  in  future  systems,  and  new  training 
practices. 

On  a  more  practical  level,  planning  decisions  are  best 
performed  with  the  cooperation  of  representatives  from 
the  prospective  funding  agencies  and  are  most  likely  to 
be  implemented  when  these  agencies  request  such  service. 
Moreover,  it  is  not  likely  that  any  single  plan  would 
meet  the  needs  of  all  federal  agencies  concerned  with 
simulation.  The  best  course  of  action  we  see  is  for  the 
various  agencies  concerned  with  simulation  to  identify 
their  specific  needs  and  then  coordinate  with  each  other 
to  promote  complementary  research  and  avoid  duplication 
of  effort. 

A  systematic  look  to  the  future  is  essential  to 
planning  simulation  research.  Too  often  the  approach  has 
been  to  use  existing  concepts,  equipment  capability,  and 
facilities  as  a  basis  for  behavioral  science  research  in 
simulation.  Newer  roles  for  humans,  anticipated  hardware 
capability,  evolving  uses  of  simulation,  and  advanced 
training  concepts  should  be  considered  so  that  research 
can  be  in  a  better  position  to  influence  future  needs. 
Although  the  recommendation  that  changing  roles  for  humans 
in  systems  and  newer  training  concepts  and  equipment 
technology  should  be  identified  as  a  deliberate  part  of 
research  planning  may  seem  obvious,  it  is  not  typically 
followed. 

We  describe  some  of  the  types  of  projections  regarding 
future  directions  that  may  be  of  use  in  research  planning 
below: 
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1.  The  shifting  role  of  the  human  in  systems  from 
system  operator  and  continuous  controller  to  supervisor 
can  change  the  fundamental  characteristics  of  simulators 
and  how  they  are  used.  Newer  operational  roles  will 
emphasize  high-level  decision  making  as  in  multi-operator 
command-and-control  systems  and  advanced  computer-driven 
aircraft.  The  operator  can  be  either  local  or  remote  as 
well  as  have  direct  or  indirect  control. 

2.  Computer  science  and  technology  at  lower  relative 
cost  will  continue  to  influence  the  characteristics  of 
simulators.  Particular  areas  in  which  capabilities  will 
increase  include  high-quality,  interactive  visual  scene 
generation,  integration  of  complex  system  and  mission 
elements  either  as  single  or  federated  devices,  and 
low-cost  stand-alone  devices. 

3.  Concepts  such  as  cognitive  skill  development, 
artificial  intelligence,  and  embedded  training  as  well  as 
capabilities  such  as  computer-assisted  and  computer- 
managed  training,  microprocessor -based  part-task  devices, 
and  telecommunications  for  network  linking  of  simulators 
should  be  considered  in  research  planning  as  ways  of 
increasing  the  utility  of  simulators. 

The  working  group  did  not  attempt  to  develop  a  long- 
range  comprehensive  research  plan  during  this  study, 
since  it  was  not  within  our  scope  to  do  so.  However,  we 
developed  an  integrated  series  of  specific  recommendations 
that  constitute  the  major  components  of  a  comprehensive 
plan  (see  Chapter  5) . 

2.  Long-Range  Stable  Funding  Should  Be  Provided  to 
Encourage  the  Development  of  Academic  Bases  for 
Simulation  Research. 

Funding  should  be  provided  for  long-range  programs  of 
research  on  fundamental  behavioral  issues  relevant  to 
simulation.  The  principal  object  of  this  funding  should 
be  to  foster  the  development  of  a  coherent  science  and 
technology  of  simulation  at  academically  based  centers  of 
excellence.*  A  second  purpose  would  be  to  support 
research  in  other  settings  such  as  government  and  private 
laboratories. 


*Since  this  recommendation  was  written,  Florida  Central 
University  has  established  an  Institute  for  Simulation 
and  Training. 
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A  recent  DoD  study  (Herman,  1982)  identified,  among 
other  things,  the  need  for  fundamental  research  programs 
to  improve  the  design  and  use  of  simulators  and  recom¬ 
mended  that  defense  centers  of  excellence  in  the  Depart¬ 
ment  of  Defense  be  established  in  several  multidiscipli¬ 
nary  areas  including  simulation.  We  believe  that  academic 
bases,  complemented  by  industry  and  government  research, 
are  preferable  because  they  fulfill  as  well  the  important 
traditional  role  of  educating  scientists  and  engineers 
and  serving  as  focal  points  for  the  accumulation,  integra¬ 
tion,  and  dissemination  of  knowledge.  Academic  institu¬ 
tions  also  are  less  apt  to  be  subject  to  rapidly  changing 
priorities,  thus  contributing  to  long-term  stability  and 
continuity  in  the  research  programs. 

This  is  not  to  say  that  currently  existing  government 
programs  and  facilities  for  simulator  research  are  not 
effective,  but  rather  that  they  are  both  limited  in 
capacity  and  scope  relative  to  the  amount  and  kind  of 
research  required  and  adequate  to  meet  the  educational  or 
information  communication  needs.  These  needs  are  ful¬ 
filled  as  a  matter  of  course  and  direct  consequence  of 
the  existence  of  research  programs  at  academic  institu¬ 
tions. 

Ideally,  academic  bases  for  simulation  would  become 
national  multidisciplinary  centers  for  research  and 
development  in  the  science  and  technology  of  simulation, 
with  human  resources  of  high  quality  in  the  behavioral 
and  computer  sciences  as  well  as  engineering.  These 
centers  would  provide  a  focused  research  program  on 
problems  of  simulation,  encourage  formation  of  inter¬ 
disciplinary  groups  of  research  scientists  and  inter¬ 
disciplinary  training  programs  focused  on  simulation,  and 
attract  qualified  students  who  are  prepared  upon  gradua¬ 
tion  to  contribute  to  research  on  and  development  of 
simulation  and  its  use.  ttiese  centers  would  also  be  a 
resource  for  investigators  from  other  institutions  to 
visit  and  collaborate  on  research  and  development 
projects. 

The  educational  role  of  these  academic  bases  would 
provide  training  at  both  undergraduate  and  graduate 
levels  for  both  practitioners  and  basic  researchers.  The 
purpose  would  be  to  train  individuals  in  simulation 
science  to  teach  in  other  universities,  perform  funda¬ 
mental  and  applied  research  in  government,  industry,  and 
universities,  and  apply  their  knowledge  to  simulator 
development  and  use. 


However,  such  programs  cannot  begin  without  the  incen¬ 
tive  of  adequate  funding  being  provided  initially  and  the 
prospect  of  its  continued  availability.  The  effort  to 
foster  a  coherent  science  of  simulation  with  a  strong 
academic  base  requires  a  substantial  and  long-term  invest¬ 
ment  that  is  coordinated  to  meet  the  needs  of  government 
agencies  and  industry. 

We  envision  the  establishment  of  academic  centers 
through  the  process  of  competition  for  research  funds 
based  on  the  quality  of  the  institutions,  their  faculties, 
and  the  presence  and  strength  of  behavioral  science,  engi¬ 
neering,  and  computer  science  departments,  which  are  the 
foundations  for  the  science  and  technology  of  simulation. 
Government  agencies  should  invite  university  departments 
and  interdisciplinary  programs  to  submit  proposals  for 
grants  to  support  scientific  research  and  development  in 
simulation.  They  also  should  include  support  for  faculty 
that  would  enable  development  of  new  programs  emphasizing 
scientific  training  in  simulation  problems  and  support 
for  educational  costs  of  students  as  well  as  internships 
and  postdoctoral  fellowships  both  for  beginning  and 
experienced  scientists  at  active  simulator  facilities. 

An  important  outcome  of  academic  bases  for  simulation 
research  would  be  the  development  and  dissemination  of 
integrated  information  bases  in  the  form  of  journal 
articles,  collections  of  review  papers,  and  books  on 
topics  central  to  simulation.  There  are  currently  no 
complete  and  accessible  libraries  of  relevant  documents. 
Sources  are  far-flung  and  many  documents  have  been  lost 
or  have  limited  availability.  As  a  result,  there  has 
been  much  repetition  of  work  in  the  past  three  decades. 
There  are  few  single  sources  of  quality  information  in 
the  field.  One  exception  is  the  book  Man-Machine  System 
Exper iments  (Parsons,  1972),  which  documents  simulator- 
based  research  on  command  and  control  systems.  Books 
that  evaluate  and  integrate  existing  work  are  clearly 
needed  in  areas  such  as  human  performance  measurement, 
visual  systems,  training  use,  and  design  use  of 
simulators. 

Facilities  and  equipment  would  be  a  major  consideration 
in  the  establishment  of  academic  bases  for  simulation 
research.  We  have  deliberately  avoided  addressing  the 
issue  of  facilities.  Simulation  research  can  be  expensive 
and  time-consuming  and  cannot  be  conducted  only  in  tradi¬ 
tional  laboratory  settings,  while  it  is  important  that 
academically  based  research  be  directed  to  the  development 
of  principles  that  can  be  generalized  to  specific  appli- 
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cation  needs,  it  is  not  clear,  and  by  no  means  uncontro- 
versial,  that  large-scale  simulation  facilities  located 
at  the  academic  bases  are  essential  for  this  purpose. 

On  one  hand,  it  is  true  that  fundamental  concepts  can 
be  explored  and  developed  and  educational  purposes  served 
with  reasonably  modest  equipment;  on  the  other  hand,  the 
complexity  of  operational  systems,  task  requirements,  and 
environments  and,  hence,  simulators,  can  have  major 
influences  on  performance  and  learning.  Access  to  complex 
simulation  equipment  as  well  as  to  operational  systems 
such  as  nuclear  power  plants,  aircraft,  and  command  and 
control  centers  will  be  needed  for  the  development  of 
criteria  and  validation. 

Simple  studies  have  limited  potential.  That  is,  the 
more  abstract  the  research  context  is  relative  to  the 
application  context,  the  more  uncertain  it  is  that  per¬ 
formance  and  learning  effects  found  in  research  settings 
will  obtain  in  the  application  context.  This  implies 
that  final  validation  of  research  must  occur  in  simulation 
settings  at  least  approaching  the  complexity  of  actual 
systems. 

Large-scale  simulation  facilities  are  expensive  not 
only  to  purchase  but  also  to  maintain,  change,  and  up¬ 
grade.  And  no  single  type  of  simulator  will  serve  all 
purposes.  Clearly,  a  flight  simulator  cannot  be  used  for 
investigations  of  all  issues  relevant  to  a  power  plant 
simulator.  At  some  point  the  unique  character  of  fun¬ 
damentally  different  systems  must  be  taken  into  account. 
Whether  funding  agencies  could  justify  the  cost  of  main¬ 
taining  several  large  simulators,  even  if  each  is  at  a 
different  location,  is  questionable.  However,  just  as 
full  mission  training  simulators  need  not  be  used  for  all 
stages  of  training,  all  research  need  not  be  conducted  in 
complex,  expensive  simulators.  The  most  expedient 
approach  is  to  fund  the  development  of  relatively  modest 
research  simulation  facilities  at  academic  bases  initially 
and  let  justification  for  more  expensive  devices  depend 
on  the  importance  of  the  research  product. 

Finally,  academic  research  programs  could  be  enhanced 
by  access  to  the  more  comprehensive  simulators  that  exist 
at  government  and  industrial  facilities.  The  unique  role 
of  each  type  of  organization  should  be  recognized.  The 
academic  institution  should  focus  on  fundamental  research, 
teaching,  and  information  integration.  Government  and 
industrial  simulation  laboratories  are  best  suited  to 
perform  applications  research,  validate  principles 
developed  through  academic  research,  and  ultimately  do 


the  developmental  work  necessary  for  final  implementation 
in  hardware  and  operations. 


3 .  Research  to  Develop  Near-Real-Time  Human  Performance 
Assessment  Capability  for  Simulation  is  Urgently  Needed. 

Effective  measurement  is  the  cornerstone  for  cost- 
effective  design  and  better  use  of  simulators.  Further 
progress  will  be  limited  without  significant  advances  in 
human  performance  measurement  capability.  A  focused, 
well-funded,  and  long-term  effort  is  needed  to  develop 
measurement  techniques  that  are  meaningful,  quantitative, 
and  economical  to  use  and  that  provide  data  in  a  usable 
format  on  a  timely  basis.  Recent  advances  in  computer 
and  software  technology  make  such  advances  feasible. 

Performance  measurement  development  is  a  demanding 
activity  that  should  involve  several  disciplines, 
including  systems  engineering  and  analysis,  experimental 
psychology,  computer  science,  and  statistics.  Collection 
of  data  alone  is  not  performance  measurement.  Measurement 
implies  a  definition  of  the  performance  to  be  measured, 
criteria  of  good  and  bad  performance,  the  selection  of 
data  sources,  weighting  and  integration  of  data  from  these 
various  sources  to  form  a  measure,  and  interpretation  of 
the  measures  in  terms  relevant  to  the  simulator  applica¬ 
tion. 

Further  development  of  performance  measurement  capa¬ 
bilities  for  simulator  applications  depends  on  research 
to  develop  useful  conceptions  of  the  covert  behavioral 
processes  that  lead  to  overt  actions.  Such  conceptions 
are  essential  to  define  sufficient  and  necessary  per¬ 
formance  measures  but  are  likely  to  be  a  long  time  in 
coming.  More  immediate  but  limited  benefits  could  be 
realized  by  research  to  establish  systematic  methods  for 
development  of  empirically  based  performance  measures 
that  do  not  depend  on  conceptions  of  internal  behavioral 
processes  but  are  able  to  reliably  distinguish  among 
categories  of  performance  such  as  stages  of  learning  or 
levels  of  proficiency. 

The  developments  should  encompass  the  multiple  uses  of 
measurement  in  simulation  to  include: 

1.  Feedback  during  training; 

2.  Proficiency  level  assessment  and  performance 
prediction; 

3.  Research  on  fundamental  and  applied  problems, 
including  the  effects  of  simulator  characteristics 
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on  short-  and  long-term  retention  proficiency  for 
representative  tasks; 

4.  Assessment  of  system  capability  during  design, 
development,  test,  and  evaluation;  and 

5.  Reliable  and  valid  assessment  of  capability  for 
licensing  and  certification  purposes. 

Whether  theoretically  or  empirically  based,  performance 
measurement  for  simulator  applications  must  be  implemented 
in  a  fashion  to  obtain  useful  outputs  on  a  near-real-time 
basis.  The  need  for  near-real-time  measurement  is  criti¬ 
cal.  Most  of  the  above  uses  require  data  during  or  soon 
after  a  simulator  session.  Even  in  the  case  of  research, 
data  reduction  and  analysis  become  so  ponderous  and 
delayed  that  they  reduce  considerably  the  potential 
utility  of  the  research.  Work  is  needed  on  methods  to 
automate  all  phases  of  the  performance  measurement  process 
from  data  collection  to  final  summarization  in  order  to 
take  full  advantage  of  simulator  capabilities. 

The  approach  to  development  should  be  interdisciplinary 
and  include  psychometrics,  software,  hardware,  and  opera¬ 
tional  considerations.  Compatibility  should  be  sought 
with  measurement  systems  planned  for  comparable  opera¬ 
tional  equipment  to  permit  cross-validation  and 
follow-through  assessments . 

The  broad  context  of  measurement  systems  should  be  con¬ 
sidered,  including  the  complementary  interrelationship  of 
numerical,  rating  scale,  audio/video  recording,  and 
physiological  approaches.  In  addition, , operationally 
based  performance  criteria  are  necessary  and  their 
development  should  be  given  high  priority. 

Research  on  performance  measurement  development  should 
have  both  fundamental  and  application  aspects.  The  basic 
core  should  consider  concepts,  techniques,  classes  of 
tasks,  and  the  relation  to  modeling.  The  application 
part  should  be  directed  to  specific  classes  of  problems 
such  as  developing  criteria,  converting  raw  data  into 
measures,  and  implementing  the  measurement  systems  into 
simulators  and  possibly  embedding  them  in  operational 
equipment. 
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DISSENT 
Jesse  Orlansky 

I  do  not  agree  with  General  Recommendation  1,  that 
long-range,  comprehensive  and  forward-looking  research 
plans  should  be  developed  to  address  persistent  and 
emerging  simulation  problems.  The  working  group  did,  in 
fact,  recommend  research  on  a  number  of  specific  issues 
that  include  performance  measurement,  cognitive  skills, 
failure  and  out-of-tolerance  conditions,  training  simu¬ 
lator  design  guides,  a  training  simulator  instructor 
guide,  stress  in  simulator  training  and  testing,  and 
improved  task  analysis  methodology.  I  agree  with  these 
recommendations.  I  see  no  reason  to  recommend,  in 
addition,  that  long-range,  comprehensive,  and  forward- 
looking  research  plans  be  developed  to  address  persistent 
and  emerging  simulation  problems  until  we  see  some  of  the 
fruits  of  the  research  we  have  already  recommended.  More¬ 
over,  I  believe  that  a  general  recommendation  to  develop 
forward-looking  research  plans  is  too  vague  to  be  useful 
to  any  funding  agency.  It  is  a  fact  that  our  working 
group  considered  the  possibility  of  doing  this  and  decided 
that  we  did  not  have  sufficient  time.  I  personally 
believe  that  we  did  not  grasp  the  opportunity  because  we 
did  not  have  anything  useful  to  say  about  forward-looking 
research  plans  beyond  what  appears  in  the  report. 

I  do  not  agree  with  General  Recommendation  2,  that 
long-range,  stable  funding  should  be  provided  to  encourage 
the  development  of  academic  bases  for  simulation  research, 
primarily  because  we  have  not  addressed,  even  in  general 
terms,  what  the  academic  bases  should  be  encouraged  to  do. 
The  recommendation  calls  for  academic  bases,  joint  work 
by  several  departments  at  more  than  one  school,  large- 
scale  simulation  research  facilities,  and  complementary 
support  of  industry  and  government  in  related  areas. 

This  is  an  exhilarating  prospect.  In  our  version,  the 
recommendation  proposes  to  spend  large  amounts  of  money 
to  achieve  obscure  ends.  There  may  be  a  good  idea  buried 
here,  but  it  needs  more  development. 


APPENDIX  A 


EXAMPLES  OF  SIMULATORS 


In  this  appendix  we  present  examples  of  specific 
simulators  in  each  of  five  categories:  (1)  engineering 
simulators,  (2)  training  simulators,  (3)  research 
simulators,  (4)  battle  simulators,  and  (5)  embedded 
simulation.  The  specific  examples  of  simulators  des¬ 
cribed  within  each  category  were  chosen  because  they  were 
good  illustrations  of  one  or  more  major  aspects  of 
simulator  characteristics. 

In  the  category  of  engineering  simulators,  the  aircraft 
mission  simulator  and  the  part-task  design  simulator 
illustrate  the  complexity  of  simulators,  ranging  from 
multistation  facilities  to  a  functional  representation  of 
a  specific  subsystem. 

In  the  category  of  training  simulators,  three  kinds  of 
simulators  are  described:  a  flight  training  simulator 
and  two  kinds  of  maintenance  simulators.  The  description 
of  the  flight  training  simulator  illustrates  that  instruc¬ 
tional  support  features  are  important  components  of  the 
simulator.  That  is,  there  is  more  to  a  training  simulator 
than  just  representation  of  an  item  of  equipment  or  sys¬ 
tem.  The  description  of  the  first  type  of  maintenance 
simulator  (actually  two  versions  of  a  simulator  for  the 
same  purpose)  illustrates  that  functional  representations 
of  a  simulator  can  be  quite  disparate  in  cost  and  degree 
of  physical  similarity  to  the  actual  equipment  but  equally 
effective  for  training.  The  second  type  of  maintenance 
training  simulator  described  illustrates  that  a  simulator 
can  be  a  conceptual  rather  than  physical  representation 
of  a  system  and  underscores  the  point  (described  in 
Chapter  3)  that  attaining  behavioral  objectives  and  not 
physical  emulation  is  the  principal  purpose  of  simulators. 

In  the  category  of  research  simulators,  the  ship 
bridge  simulator  described  illustrates  the  use  of 
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simulators  to  perform  fundamental  research  on  behavioral 
and  human  factors  issues  and,  for  research  purposes,  that 
special  capabilities  for  experimental  control  and  per¬ 
formance  measurement  are  essential  to  the  intended 
purpose. 

In  the  category  of  battle  simulation,  the  examples 
illustrate  several  different  points:  the  use  of  multiple 
small  weapons  simulators  in  the  context  of  field  exer¬ 
cises;  command-and-control  simulation,  in  which  the 
essence  of  the  simulation  is  information  flow;  and  the 
fact  that  scenarios,  rather  than  representations  of 
physical  equipment,  are  the  predominant  features. 

In  the  category  of  embedded  simulation,  the  two 
examples  described  highlight  the  fact  that  simulation  can 
be  part  of  operational  systems,  and  training  can  be 
extended  to  the  operational  context. 

In  addition  to  characterizing  the  diversity  of  simu¬ 
lation  types  and  uses,  there  is  a  secondary  theme  running 
through  this  appendix:  the  characteristics  of  simulators 
differ  according  to  their  behavioral  purposes.  While 
every  simulator  represents  a  particular  system  or  piece 
of  equipment,  the  comprehensiveness  and  form  of  the 
representation  can  be  very  different,  not  so  much  as  a 
function  of  the  nature  of  the  system  as  the  behavioral 
objective — that  is,  what  the  participant  is  supposed  to 
do. 

A  final  point  that  is  important  to  note  is  that  the 
examples  of  various  uses  and  types  of  simulators  are 
selective;  flight  simulators  are  emphasized,  and  ground 
vehicles  and  environmental  simulators  are  not  included. 
This  selectiveness  reflects  the  fact  that  flight  simu¬ 
lation  has  been  and  continues  to  be  the  major  focus  for 
design,  training,  and  research  uses  of  simulators;  more 
has  been  written  about  them  than  any  other  type  of 
simulator.  Ground  vehicle  simulators  for  automobiles, 
trucks,  tanks,  and  locomotives,  while  showing  important 
uses  for  training  and  research,  generally  reflect  the 
technology  used  in  other  simulation  applications. 
Environmental  simulators  such  as  centrifuges,  water 
tanks,  hot  and  cold  rooms,  hyperbaric  and  hypobaric 
chambers,  and  acoustical  chambers  are  more  closely 
associated  with  physiological  uses  than  behavioral  uses 
and  we  have  excluded  them  from  consideration  to  keep  this 
study  to  manageable  proportions. 

It  became  apparent  during  our  detailed  examination  of 
simulation  that  there  were  many  common  human  factors 
issues  that  cut  across  the  various  types  and  uses.  Table 
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A-l  illustrates  this  point  with  the  issues  specific  to 
particular  uses. 


ENGINEERING  SIMULATORS 

Simulators  have  become  an  indispensable  engineering 
tool  in  the  design,  development,  and  evaluation  of  complex 
military  and  civilian  systems.  Particularly  in  the  design 
of  new  computer-driven  multifunction  controls  and  dis¬ 
plays,  radical  departures  in  concepts  have  been  achieved. 
Engineering  simulators  are  valuable  because  quasi- 
experimental  data  can  be  obtained  on  crew  performance  for 
complex  tasks  within  the  time  frame  dictated  by  design 
and  development  schedules.  They  have  been  used  exten¬ 
sively  since  the  1940s  for  aircraft  systems;  in  the  1950s 
and  1960s  they  came  into  use  for  other  systems  such  as 
command,  control,  and  communication  (C^) ,  weaponry,  and 
space  vehicles.  The  complexity  of  modern  systems  and 
"fly  before  buy"  approaches  to  procurement  also  give 
impetus  to  the  use  of  simulation  for  these  purposes. 

Some  of  the  best  examples  of  engineering  simulators 
are  the  interactive,  high-fidelity  military  aircraft 
simulators  that  allow  performance  of  critical  mission 
segments  in  multicrew,  multivehicle  environments.  They 
can  be  used  during  each  design  stage  from  early  concept 
phases  through  testing  and  operations. 

Aircraft  engineering  simulators  support  design  in 
several  ways,  including  integration  of  avionics;  devel¬ 
opment  of  logic,  software,  procedures,  and  tactics; 
design  of  controls  and  displays;  and  evaluation  of  sub¬ 
system  and  system  performance.  Pilots  serve  as  subjects 
to  contribute  to  and  verify  human  engineering  design  in 
terms  of  workload,  operational  suitability,  and  estimates 
of  training  requirements. 

Engineering  development  simulators  include  part-task 
devices  for  single  critical  functions  that  may  be  isolated 
such  as  target  acquisition,  multiple-task  devices  such  as 
procedures  trainers  for  crew  station  configuration 
studies,  and  full  mission  devices  with  visual  attachments 
for  examining  mission  tasks  such  as  air-to-ground  weapon 
delivery.  Motion-base  devices  are  used  when  appropriate, 
most  typically  for  handling  studies  in  transport  and 
V/STOL  aircraft,  catapulting,  and  adverse  vibration  or 
oscillation  during  low-altitude,  high-speed  flight. 

Simulation  has  been  used  very  successfully  at  Bell 
Laboratories  for  design  and  development  of  human-machine 
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TABLE  A-l  Common  and  Unique  Issues  for  Simulator 
Applications 

Simulator  Use  or  Type 


Training 

Battle 

Embedded 

Key  Issues 

Engineering 

Design 

Research  Simulation 

Simulation 

Economic# 

X 

X 

X 

experimental# 
and  test  design 

Models  of  human 

X 

X 

X  . 

X 

X 

performance 

Performance 

X 

X 

X 

X 

X 

measurement 

Fidelity 

X 

X 

X 

X 

X 

requirements 
Control  and 

X 

X 

X 

X 

support  methods 
and  features 

More  refined  X 

task  analysis 

Selection  and  X  X 

character izat ion 
of  subjects 

Flexible  designs,  X  X 

procedures,  and 
performance 
measurement 

Efficient  scenario  X 

construction 

Methods  for  X  X 

translating 
training 
requirements 
into  equipment 
and  course  features 


Transition  from  X 

operational  to 
training  status 

Level  at  which  X 

simulation  occurs 

Added  hardware  X 

and  software 

requirements  X 
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systems  in  equipment  related  to  communication  systems 
(Holt  and  Stevenson,  1977) .  Bell  Laboratories  developed 
the  concept  of  functional  design  of  an  automated  direc¬ 
tory  assistance  system,  including  operating  procedures 
and  operator  aids,  in  simulation  prior  to  initiating 
full-scale  engineering  development.  A  computer-based 
message-switching  system  was  also  designed  through 
simulation  concurrently  with  engineering  development. 
Both  simulations  were  successful  and,  in  addition, 
documentation  for  operating  procedures  and  maintenance 
and  training  time  were  reduced. 

We  present  below  examples  of  engineering  design  and 
development  simulators.  Representative  devices  are  des¬ 
cribed  in  the  areas  of  avionics  and  battle  simulation. 
Avionics  systems  are  used  to  illustrate  the  use  of 
simulation  in  the  development  of  military  platforms; 
descriptions  of  simulators  for  systems  such  as  tanks, 
aircraft  carriers,  submarines,  and  space  platforms  would 
be  quite  similar. 


Aircraft  Mission  Simulators 

These  devices  usually  represent  the  state  of  the  art 
in  simulation  technology.  Simulator  cockpits  are  like  a 
set  of  building  blocks;  configurations  can  be  modified  as 
system  design  progresses.  Because  of  their  flexibility 
they  can  be  changed  readily  and  allow  high-fidelity  dupli¬ 
cation  of  critical  crew  tasks.  A  new  cockpit  can  be  put 
in  place  in  several  hours  and  software  changes  made  to 
accommodate  an  evolving  design. 

One  of  the  most  complex  simulators  is  the  McDonnell 
Aircraft  Company's  Flight  Simulation  Facility,  shown  in 
Figure  A-l  (Jones,  1979).  This  simulator  has  been  used 
in  the  development  of  the  F-15,  F/A-18,  and  AV-8B  air¬ 
craft.  It  has  five  domes  within  which  removable  cockpits 
can  be  placed.  These  spheres  completely  enclose  the  crew 
station  and  provide  a  projection  surface  for  a  pictorial 
image  of  the  earth  and  sky.  It  has  been  used  to  define 
preliminary  crew  functions  and  operating  procedures,  to 
examine  hardware  and  display  symbology,  to  develop 
operational  doctrine  and  tactics,  and  to  establish  crew 
performance  and  workload  levels  necessary  to  meet 
operational  requirements. 

An  iterative  process  is  used  to  develop  a  new  crew 
station.  Analytical  results  are  verified  empirically  in 
the  context  of  mission  scenarios.  Normal  operations  and 
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tactical  situations  are  addressed  with  low  and  high  work¬ 
loads  imposed  as  test  conditions.  Numerical  and  graphic 
data  and  video  recordings  and  detailed  pilot  debriefings 
are  obtained  for  each  mission.  Pilots  are  given  detailed 
questionnaires  that  focus  on  problem  areas  and  design  and 
operating  solutions. 

Preliminary  studies  are  conducted  with  test  pilots  and 
engineers  from  _he  fields  of  guidance  and  control,  avi¬ 
onics,  aerodynamics,  flight  test,  computer  software, 
operations  analysis,  crew  station  design,  and  human 
factors.  Selected  service  pilots  participate  as  test 
subjects  and  operational  experts  for  design  verification. 


Part-Task  Design  Simulators 

An  illustrative  use  of  a  part-task  simulator  is  the 
simulation  of  target  acquisition  tasks  using  an  infrared 
imaging  sensor.  A  McDonnell  Douglas  Astronautics  Company 
simulator  provides  relatively  high-fidelity  reproductions 
of  a  missile's  infrared  target  acquisition  system. 

Cathode  ray  tube  (CRT)  displays  are  generated  with  a 
bench-mounted  optical  train,  television  camera,  servo- 
controlled  zoom  lens,  X-Y  transport  for  movement  of  the 
target  imagery,  and  a  video  mixer.  Target  materials  are 
prepared  for  various  ship  types  and  orientations  by 
photographing  special  models,  although  real  imagery,  if 
available,  can  be  used.  Control  of  starting  range,  zoom 
rate,  contrast  level,  and  display  sequencing  are  computer 
controlled.  Operators  register  their  decisions  by 
pressing  control  buttons.  Performance  data  are  recorded 
and  stored  in  the  computer. 

The  part-task  simulation  of  the  imaging  infrared 
(12R)  system  for  ship  target  acquisition  was  used  to 
determine  feasibility  of  the  concept,  develop  design  data 
to  optimize  system  performance,  and  estimate  the  capabil¬ 
ity  of  the  system  to  acquire  designated  military  targets 
under  a  variety  of  operational  conditions  (Mocharnuk  et 
al.,  1981).  Criterion  measurements  were  obtained  in  terms 
of  classification,  range,  and  accuracy.  Some  variables 
considered  included  ship  type,  aspect  angle,  display  con¬ 
trast,  frame  rate,  field  of  view,  and  the  use  of  cueing 
aids. 
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TRAINING  SIMULATORS 

Training  is  undoubtedly  the  most  common  use  of  simu¬ 
lators.  A  training  simulator  is  an  interactive  device  to 
facilitate  the  acquisition  of  skills  and  knowledge. 
Simulators  are  used  for  initial  and  continuation  training 
of  operators,  maintenance  personnel,  and  command  and 
supervisory  personnel.  Devices  duplicating  or  represent¬ 
ing  all  or  part  of  actual  equipment  or  systems  are  used 
extensively  for  such  purposes  as  familiarization,  acquir¬ 
ing  procedural  and  continuous  control  skills,  learning 
part-  or  whole-task  performance,  and  the  practice  of  seg¬ 
ments  of  or  complete  missions.  The  largest  use  of  simu¬ 
lation  is  for  individual  training,  although  some  devices, 
such  as  a  B-52  simulator  or  MILES,  support  training  of 
crews,  teams,  and  even  entire  units. 

A  simulator  that  duplicates  the  function  of  an  item  of 
equipment  or  system  can  be  used  for  training  purposes  in 
much  the  same  way  as  the  actual  item.  However,  most 
training  simulators  incorporate  instructional  features  to 
facilitate  and  control  the  training  process.  How  a  simu¬ 
lator  is  used  and  the  availability  of  instructional  sup¬ 
port  features  can  be  as  important  as  the  simulation 
characteristics  of  the  device.  A  simulator  does  not 
instruct;  it  is  only  a  means  to  implement  and  support  a 
training  program. 

Instructional  support  features  usually  include,  as  a 
minimum,  a  situational  context  or  scenario,  the  means  for 
instructional  control  of  the  simulation,  providing  per¬ 
formance  feedback  to  the  trainee,  and  supplying  per¬ 
formance  information  to  the  instructor.  Implementing 
simulator  instructional  features  may  require  a  small  or 
large  amount  of  additional  hardware  and  software.  For 
example,  a  rifle  simulator  may  require  nothing  more  than 
a  means  for  displaying  the  location  of  target  hits.  The 
target,  representing  the  enemy,  is  the  situational 
context;  the  instructor  observes  the  performance  of  the 
trainee  directly,  both  instructor  and  trainee  receive 
performance  feedback  from  the  target  display,  and 
training  is  controlled  by  verbal  communication. 

For  a  tactical  command-and-control  or  full-mission 
flight  training  simulator,  the  scenario  is  resident  in  a 
rather  complex  computer  program.  The  instructor  may  sit 
at  a  separate  console  that  allows  him  to  start  and  stop 
the  scenario,  inject  certain  conditions,  and  view  a  dis¬ 
play  that  provides  summary  information  about  the  actions 
taken  by  the  student  and  their  effects  on  the  rest  of  the 
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scenario.  The  student  may  receive  feedback  about  the 
consequences  of  his  actions  both  directly  from  a  situa¬ 
tion  display  and  from  certain  display  augmentation 
features,  which  show  the  ultimate  consequences  of  the 
actions  or  an  immediate  evaluation  of  their  appropriate¬ 
ness. 

We  describe  three  examples  of  training  simulators: 
the  first  is  a  flight  simulator,  and  the  second  two  are 
maintenance  simulators,  one  for  a  steam  propulsion  system 
and  the  other  for  avionics  electronics  maintenance. 


Plight  Training  Simulators 

Device  2B24  is  a  helicopter  simulator  for  the  training 
of  instrument  flight  for  the  UH-1  series  of  utility  heli¬ 
copters.  The  device  consists  of  four  cockpits,  each 
mounted  on  its  own  f ive-degrees-of-freedom  motion  base 
and  controlled  by  a  single  computer  system.  The  appear¬ 
ance  and  function  of  the  cockpit  instruments  and  controls 
are  identical  to  the  UH-1  helicopter. 

Device  2B24  was  designed  with  emphasis  on  training 
rather  than  simply  reproducing  the  relatively  inefficient 
learning  environment  of  an  actual  helicopter.  Its  con¬ 
figuration  and  scope  of  functions  were  based  on  an 
analysis  of  training  objectives  and  processes.  Special 
displays  for  the  instructor,  located  both  in  the  cab  of 
each  cockpit  and  at  the  central  operator  station,  display 
the  position  within  the  gaming  area,  the  location  of 
radio  navigational  aids,  and  time  history  profiles  of 
altitude  and  airspeed. 

As  an  instrument  trainer,  the  simulator  has  no  external 
visual  system  but  does  have  a  sound  system  to  portray 
rotor  noises  that  change  with  speed  and  load.  Simulated 
flight  can  occur  within  a  100-by-100-mile  gaming  area, 
which  is  a  representation  of  the  local  environment  in 
which  the  simulator  is  located.  All  radio  navigational 
aids  that  exist  in  the  represented  area  are  included  in 
the  simulation. 

Partially  automated  instructor  functions  include  the 
demonstration  of  maneuvers,  sequencing  of  instruction, 
varying  task  difficulty,  monitoring  performance,  updated 
scenarios,  and  extensive  capabilities  for  reprogramming 
training  situations.  Feedback  to  the  student  during 
training  is  possible  via  real-time  and  slow-time  playbacks 
of  the  student's  control  performance,  video  and  audio 
recording  and  playback  systems,  on-line  displays  sum- 
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marizing  student  performance  over  time,  and  hard  copy 
printout  of  performance  data.  Provision  is  also  made  for 
students  to  exercise  control  over  their  own  training. 

Device  2B24  may  be  used  in  any  of  three  modes,  semi¬ 
automatic  ( SEMI -AUTO ) ,  automatic  (AUTO),  and  checkride 
(CK-RIDE) .  In  the  SEMI-AUTO  mode,  the  aircraft  initially 
can  be  placed  in  any  location  and  altitude  within  the 
gaming  area,  flown  anywhere  within  the  gaming  area  using 
the  flight  instruments  and  radio  navigational  aids,  and 
flown  through  instrument  approaches  to  several  airfields. 
In  the  AUTO  mode,  1  of  10  preprogrammed  instrument  flight 
maneuvers  can  be  selected  for  demonstration  or  practice. 
Computer-controlled  voice  tapes  provide  a  briefing  on  the 
flight  profile,  describe  demonstrated  maneuvers  as  they 
are  flown,  and  alert  the  pilot  to  check  various  flight 
instruments  during  practice.  In  the  CK-RIDE  mode  the  pilot 
flies  a  long  cross-country  trip  without  voice  alerts  and 
basic  variables  of  flight  such  as  altitude  and  airspeed 
are  recorded  for  printout  at  the  end  of  the  test  session. 


Maintenance  Training  Simulators 

Highly  skilled  technicians  are  required  to  provide 
servicing,  troubleshooting,  and  repair  of  malfunctions  in 
sophisticated  equipment  such  as  high  pressure  steam  pro¬ 
pulsion  systems  in  ships,  aircraft  engines,  and  the  wide 
variety  of  complex  electronic  gear  in  ground,  air,  and 
sea  systems.  Maintenance  training  simulators  have  been 
developed  to  meet  the  need  to  train  the  large  numbers  of 
maintenance  technicians  required  and  to  provide  a  capabil¬ 
ity  to  present  a  broad  range  of  maintenance  problems  to 
the  trainee.  Maintenance  trainers  frequently  cost  less 
than  actual  equipment  and  are  more  reliable,  therefore 
more  available  for  training  than  actual  equipment. 

Many  maintenance  simulators,  particularly  for  elec¬ 
tronics,  such  as  the  6883  Converter/Flight  Control 
Systems  Test  Station  for  Air  Force  F-lll  aircraft,  the 
Navy  A-7E  Head-Up  Display  Test  Bench,  and  the  Navy  MA-3 
aircraft  12  KVA  generator  Test  Bench,  have  a  high  degree 
of  resemblance  to,  and  in  some  cases  cannot  be  distin¬ 
guished  superficially  from,  actual  equipment.  The 
controls,  instruments,  displays,  and  probes  for  testing 
components  perform  as  if  they  were  actual  equipment. 
Components  under  test  respond  correctly  to  test  probes, 
i.e.,  with  all  the  symptoms  of  a  particular  malfunction. 

If  any  part  of  a  circuit  board  is  found  to  be  defective, 
the  entire  board  is  removed  and  replaced. 
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There  is,  however,  a  real  difference  inside  the 
cabinet,  where  electronic  parts  or  circuit  boards  may 
consist  only  of  photos  or  engravings  of  the  real  parts. 

The  cards  are  connected  electrically  to  the  computer  and 
respond  correctly  according  to  the  simulated  malfunction 
state  because  the  computer  controls  the  symptoms.  There¬ 
in  lies  the  basis  for  reduced  cost  and  "ruggedizing"  of 
the  device  for  training  purposes. 

Maintenance  simulators  have  also  been  developed  that 
are  greatly  reduced  in  physical  fidelity.  These  rely 
mainly  on  schematics  presented  on  a  panel  board  together 
with  functional  controls,  switches,  test  points,  and 
instruments.  Some  very  new  simulators  consist  solely  of 
schematic,  functional  diagrams  with  instruments  and 
displays  drawn  on  a  cathode  ray  tube;  the  controls  and 
displays  are  addressable  by  touching  the  face  of  the  tube 
or  through  a  cursor  control  and  keyboard.  As  with  three- 
dimensional  simulators,  many  different  malfunctions  can 
be  selected  by  the  instructor.  The  military  services 
have  procured  more  than  1,100  units  of  about  200  differ¬ 
ent  two-dimensional  maintenance  simulators  since  1972. 

Some  examples  of  two-dimensional  simulators  are:  F-16 
Maintenance  Trainers  (for  avionics,  electrical,  propul¬ 
sion,  hydraulic,  and  weapons  control  systems  of  the  Air 
Force  F-16  aircraft) ,  the  Navy  generalized  Electronic 
Equipment  Maintenance  Trainer  (EEMT) ,  and  the  6883 
Converter /Flight  Control  Systems  Test  Station  for  the  Air 
Force  F-lll  aircraft. 

We  describe  two  classes  of  maintenance  simulators: 
the  first,  the  6883  Maintenance  Training  System,  consist¬ 
ing  of  both  high  and  low  physical  fidelity  versions,  is 
an  electronics  maintenance  simulator  used  to  train  people 
in  the  use  of  automated  test  equipment  for  troubleshooting 
aircraft  avionics  components.  The  second,  STEAMER,  is  a 
computer-based  symbolic  graphic  representation  of  a  steam 
propulsion  system;  it  has  little  physical  fidelity  in 
terms  of  appearance  but  is  accurate  in  its  functional  and 
relational  aspects. 


6883  Maintenance  Training  System  (6883  MTS) 

The  6883  MTS  is  a  training  simulator  for  the  Air  Force 
F-lllD  aircraft  6883  Converter/Flight  Control  Test 
Station.  This  item  of  automated  test  equipment  is  used 
in  intermediate,  "shop  level"  maintenance  to  test, 
inspect,  troubleshoot,  and  repair  faulty  line  replaceable 
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units  (LRUs)  for  the  F-111D  aircraft  multiplexer  converter 
and  flight  control  functions. 

The  6883  MTS  is  designed  to  permit  hands-on  practice 
of  the  checkout  troubleshooting  and  repair  procedures 
performed  during  maintenance  of  the  6883  itself  and  three 
associated  F-lll-D  LRUs,  the  bank  and  turn  assembly,  the 
yaw  computer,  and  the  multiplexer  converter.  The  simu¬ 
lator  consists  of  four  principle  units:  (1)  a  central 
control  computer;  (2)  an  instructor  console;  (3)  a 
student  station,  consisting  of  a  CRT,  keyboard,  and  slide 
projector  for  instructional  and  feedback  purposes;  and 
(4)  the  simulated  6883  test  station  and  LRUs. 

Both  actual  equipment  and  two  versions  of  the  6883  MTS 
have  been  used  for  training.  The  first  version  has  high 
physical  fidelity  with  respect  to  the  actual  equipment 
but  has  limited  function.  The  second  version  of  the  6883 
MTS  has  lower  physical  fidelity  than  the  actual  equipment, 
e.g.,  a  flat  panel  representation  of  the  test  station 
console,  simplified  cables  and  adapters,  and  a  schematic 
panel  representation  of  the  LRUs,  but  can  perform  all 
functions  of  the  real  test  equipment. 

The  instructor  console  permits  initial  setup  of 
lessons,  control  of  lesson  flow,  monitoring  student 
progress,  and  recording  detailed  performance  data.  The 
student  station  directs  actions  of  the  student  at  the 
test  station,  provides  immediate  feedback  on  actions 
taken,  guides  students  through  technical  material,  and 
provides  information  in  response  to  requests  for  help. 

The  test  station  and  LRUs  are  composites  of  functional 
components  and  pictorial  representations  of  other 
components  that  are  not  necessary  for  the  training 
purposes.  For  example,  some  of  the  electronic  circuit 
boards  in  the  LRUs  are  pictures,  while  others  are 
three-dimensional  representations  of  actual  circuit 
boards. 

The  6883  MTS  was  designed  to  support  training  objec¬ 
tives  but  not  to  duplicate  actual  equipment.  This 
philosophy  is  exemplified  in  the  emphasis  on  instruc¬ 
tional  support  features  that  facilitate  the  instructor's 
initiation,  control,  and  monitoring  of  training  and  that 
provide  the  student  with  easy  access  to  routine  informa¬ 
tion,  guidance,  and  feedback  that  would  normally  require 
the  presence  of  the  instructor.  Easily  understood  command 
and  feedback  statements  are  provided  by  the  student's  CRT. 
Pictorial  information  from  the  slide  projector  supplements 
technical  manual  information.  At  each  step  in  the  train¬ 
ing  sequence,  the  student  is  required  to  answer  a 


rt 


118 


multiple-choice  query  about  what  to  do  next.  This  forces 
the  student  to  think  about  each  step  and  also  provides 
error  information  to  the  instructor. 

The  overall  6883  MTS  was  ~  igned  with  hardware  and 
software  modularity  to  permit  easy  maintenance,  change, 
or  expansion  of  equipment  and  instructional  software.  A 
special  programming  language  was  developed  to  facilitate 
course  changes  or  additions.  In  other  words,  the  6883 
MTS  was  designed  with  the  inevitable  need  for  change  and 
expansion  of  function  in  mind. 


STEAMER 

The  STEAMER  project  is  an  attempt  to  develop  and 
evaluate  knowledge-based  techniques  for  use  in  portable 
computer-based  training  systems.  A  prototype  version  of 
STEAMER  has  been  field  tested  at  the  Navy's  Surface 
Warfare  Officers  School.  The  central  idea  of  the  project 
is  to  provide  a  detailed,  easily  inspected  simulation  and 
automatic  tutor  in  a  desk-top-sized  training  device.  The 
project  is  developing  techniques  for  displaying  and  con¬ 
trolling  simulation  models  and  for  automatically  pro¬ 
viding  tutorial  advice  and  explanations. 

The  project  is  focused  on  naval  propulsion  engineering 
as  a  domain  in  which  to  investigate  these  techniques. 

The  current  STEAMER  system  consists  of  a  computer-based 
simulation  of  the  propulsion  plant  of  a  1078  class  fast 
frigate,  a  dynamic  color  graphics  display  for  inspecting 
and  controlling  the  simulated  plant,  and  a  black  and 
white  display  for  exercising  other  features  of  the  system. 

A  device  called  a  mouse  is  used  to  control  a  cursor  to 
designate  items  on  the  two  STEAMER  displays.  The  student 
manipulates  the  simulated  steam  propulsion  plant  by 
designating  various  components,  such  as  valves,  and  then 
causing  them  to  open,  shut,  or  adjust  the  steam  flow. 
Changes  in  the  state  of  the  components  are  indicated  by 
changes  in  color  or  changes  on  dials,  thermometers,  and 
digital  readouts  (see  Figure  A-2) . 

This  approach  to  training  personnel  to  recognize, 
interpret,  and  adjust  to  malfunctions  and  their  impact  on 
system  performance  and  operating  procedures  is  particu¬ 
larly  useful  for  complex  systems  such  as  nuclear  power 
plants.  It  is  flexible  and  readily  adaptable  to  differ¬ 
ent  applications  through  an  appropriate  analysis  of  the 
job  to  identify  the  required  skills  and  knowledge  and 
associated  training  objectives. 
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RESEARCH  SIMULATORS 

Simulators  for  research  differ  from  simulators  for 
engineering  design  and  development  principally  in  that 
the  former  use  has  the  objective  of  gaining  generalizable 
knowledge  (i.e.,  discovering  fundamental  principles  that 
are  broadly  applicable) ,  while  the  latter  use  is  to  obtain 
specific  data  for  the  design  of  particular  items  of  equip¬ 
ment,  systems,  or  procedures.  Research  uses  of  simulators 
can  focus  on  behavioral  or  engineering  issues,  although 
these  are  frequently  interrelated.  Research  simulators 
generally  must  allow  for  greater  latitude  in  manipulating 
conditions,  events,  and  fundamental  functional  character¬ 
istics  than  engineering  simulators.  For  research, 
scenarios  do  not  necessarily  have  to  represent  actual 
situations;  often  certain  features  may  be  highly  dis¬ 
torted  or  abstracted,  events  may  be  time  compressed,  and 
secondary  tasks  may  be  created  that  have  no  real-world 
counterpart. 

There  are  a  variety  of  practical  reasons  for  the  use 
of  research  simulators,  such  as  lower  system  development 
costs,  increased  safety,  improved  efficiency  in  producing 
scenarios,  and  the  opportunity  to  obtain  otherwise 
inaccessible  measurements.  Simulators  have  been  adapted 
to  support  research  when  the  necessary  conditions  for 
experimentation  cannot  be  created  easily  in  a  conven¬ 
tional  laboratory  or  the  research  intent  focuses  on  an 
important  aspect  of  real-world  human  behavior ,  such  as 
flying  or  ship  handling.  In  addition,  the  availability 
of  obsolete  training  simulators,  particularly  flight 
simulators,  has  encouraged  university  and  other  labora¬ 
tory  investigators  to  take  advantage  of  these  devices  as 
convenient  tools  to  support  their  research. 

The  extent  of  difference  between  research  and  engineer¬ 
ing  simulators  and  their  uses  depends  a  great  deal  on  how 
basic  or  applied  the  research  is.  Clearly,  these  uses 
have  much  in  common;  both  have  the  objective  of  obtaining 
valid  and  reliable  human  performance  data,  and  the  simu¬ 
lator  characteristics,  scenarios,  and  performance 
measurement  requirements  can  be  the  same.  All  uses  of 
simulation  become  commingled  in  programs  of  research  to 
develop  design  requirements  for  training  simulators. 

We  describe  two  classes  of  simulator  facilities: 
for  aviation  and  one  for  maritime  research. 


one 
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Aviation  Research  Simulator  Facility 

The  Man-Vehicle  Systems  Research  Facility  at  NASA  Ames 
Research  Center,  Moffett  Field,  California,  is  a  good 
example  of  a  government  simulation  research  laboratory. 
This  facility  permits  work  not  feasible  in  other  existing 
NASA  flight  simulators  that  are  intended  primarily  for 
aeronautical  rather  than  human  factors  research.  Its 
general  purpose  is  the  support  of  studies  of  the  inter¬ 
actions  among  flight  crews,  their  aircraft,  and  air 
traffic  control.  The  facility  includes  a  B-727  aircraft 
simulator,  a  skeletal  cockpit  for  work  on  new  display  and 
control  designs,  and  an  air  traffic  control  center.  The 
NASA  equipment  supports  complete  simulation  of  a  flight 
mission  and  includes  a  visual  display  capable  of  depicting 
dusk  or  night  lighting,  aircraft,  fog,  clouds,  and  other 
weather  conditions.  Experimenters  can  introduce  problems 
such  as  turbulence,  air  traffic,  visual  obscurations,  and 
mechanical  failures  to  create  and  control  visual  condi¬ 
tions  as  well  as  operational  workloads. 

Much  of  the  research  focuses  on  cockpit  instrumenta¬ 
tion,  crew  procedures,  and  workload  measurement.  Topics 
of  investigations  include  how  decisions  made  in  the 
cockpit  are  affected  by  environmental  and  hardware  dif¬ 
ficulties  as  well  as  by  the  availability  of  information 
from  air  traffic  control  and  other  aircraft,  how  errors 
are  made,  and  the  effects  of  automation,  fatigue,  and 
advanced  instrumentation  on  human  performance. 

Some  special  measurement  devices  are  necessary  for  the 
particular  research  applications  of  a  flight  simulator. 
Typical  performance  data  consist  of  measures  such  as 
course  deviations,  touchdown  dispersions,  and  pilot 
control  activity.  A  great  deal  of  human  engineering 
effort  has  gone  into  the  design  and  inclusion  of  instru¬ 
ments  such  as  secondary  task  batteries,  oculometers,  and 
television  recording  systems  related  to  particular 
behavioral  and  life-science  research  interests. 

The  instrument  panels  in  many  research  simulators  may 
be  reconfigured.  The  controls,  for  example,  may  be  moved 
from  center  pedestal  to  overhead  location,  in  order  to 
reflect  differences  between  a  conventional  jet  transport 
and  a  vertical  or  short  takeoff  and  landing  (V/STOL) 
aircraft.  This  flexibility  is  achieved  at  the  expense  of 
conformity  with  the  anthropometric  and  often  the  geo¬ 
metric  designs  of  operational  aircraft. 

Use  of  general-purpose  CRT  displays  for  the  study  of 
information  display  and  control  requirements  in  advanced 
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avionics  systems  has  become  increasingly  common  over  the 
last  10  years.  In  some  cases,  panel  cutouts  with  conven¬ 
tional  instrument  identification  markings  are  placed  over 
the  CRTs  to  provide  an  aura  of  realism,  but  in  many  cases 
these  identifiers  are  merely  inserted  as  text  information. 
This  procedure  is  consistent  with  the  trend  in  the  use  of 
CRTs  in  transport  aircraft  for  area-navigation  and 
general-purpose  multifunction  displays. 

Huff  and  Nagel  (1975)  provide  an  excellent  review  of 
the  use  of  flight  simulators  for  research  as  well  as  a 
list  and  description  of  many  of  these  simulators. 


Maritime  Research  Simulator  Facility 

Since  1976  the  U.S.  Maritime  Administration  has 
operated  the  Computer-Aided  Operations  Research  Facility 
(CAORF)  to  perform  research  and  to  test  and  evaluate  ship 
operations  in  harbor  confines  and  open  sea.  The  basic 
long-range  research  program  includes  five  research  areas: 
(1)  vessel  operations  in  ports  and  waterways,  (2)  stan¬ 
dards  for  training  and  licensing,  (3)  criteria  and 
specifications  for  ship's  bridge  design,  (4)  standards 
for  watch-keeping  performance,  and  (5)  maneuvering 
response  requirements  for  ships. 

The  core  concern  in  all  these  research  areas  has  been 
on  human  factors  in  maritime  operations.  Several 
maritime  advanced  technical  developments  have  failed 
because  human  perceptual  and  control  capabilities  have 
been  neglected  when  designing  each  particular  system 
(Puglisi,  1981).  These  capabilities  are  identified  with 
the  human  limitations  in  perception,  decision  making,  and 
the  control  of  machinery.  The  main  area  in  which  these 
functions  are  undertaken  on  merchant  ships  is  on  the 
bridge.  The  duplication  of  the  bridge  environment  and 
the  modeling  of  ship  response  in  the  CAORF  provide  a 
sophisticated  human  factors  laboratory  dedicated  to 
examining  the  human  element  in  marine  operations. 

The  simulated  bridge  consists  of  a  wheelhouse  5.1 
meters  wide  and  4.3  meters  deep.  The  equipment  on  the 
bridge  is  similar  to  that  normally  available  in  the 
merchant  fleet  and  includes  steering,  controls  and 
displays  for  propulsion,  and  bow  and  stern  thrusters. 

The  two  navigation  radars  are  capable  of  displaying  both 
relative  and  true  motion  presentations  plus  collision 
avoidance  information.  Future  capabilities  will  include 
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a  digital  fathometer,  a  radio  direction  finder,  and  Loran 
C  and  Omega  global  navigation  systems. 

An  external  visual  scene  is  provided  by  a  full-color 
computer-generated  image  system,  which  depicts  ships, 
bridges,  buoys,  lighthouses,  tall  buildings,  mountains, 
etc.,  in  a  panoramic  view  covering  approximately  240 
degrees  horizontally.  The  lighting  of  the  visual  scene 
can  be  varied  continuously  from  full  sun  to  moonless 
night  and  with  any  degree  of  fog  or  haze. 

A  control  station  is  the  central  location  from  which 
the  simulator  experiment  is  controlled  and  monitored. 

The  simulation  can  be  initiated  with  the  ship  anywhere 
within  the  visual  gaming  area  and  any  traffic  configura¬ 
tion  desired.  The  control  station  enables  the  researchers 
to  communicate  with  the  watch  standing  crew  on  the  bridge, 
to  simulate  malfunctions,  and  to  control  the  operating 
mode  of  the  simulator.  It  is  also  possible  to  control 
the  motions  of  other  ships  and  tugs  in  the  gaming  area 
and  simulate  telephone,  intercom,  radio,  and  whistle 
contact  with  the  CAORF  bridge  crew.  A  separate  human 
factors  monitoring  station  is  designed  to  allow  observa¬ 
tion  of  crew  behavior.  Monitoring  is  provided  by  five 
closed-circuit  TV  cameras  and  four  microphones  appro¬ 
priately  located  throughout  the  wheelhouse  to  record  all 
activities,  comments,  and  commands. 


BATTLE  SIMULATORS 

Battle  simulation  is  one  use  of  simulators  that 
bridges  development  and  training  applications:  it  is  a 
combination  of  human-in-the-loop  simulation  and  war 
gaming.  Battle  simulator  uses  range  from  system 
development  and  evaluation  of  systems  and  tactical 
doctrine  to  training  and  the  evaluation  of  tactical 
strategies  and  operational  capabilities.  Brief 
descriptions  of  representative  human-in-the-loop  battle 
simulations  from  each  service  follow. 


Army  Battle  Simulation 

The  U.S.  Army  has  developed  two  kinds  of  battle 
simulations:  (1)  field  exercises  for  troops  operating 

over  real  terrain  and  (2)  command  post  exercises  for 
command  groups  using  simulated  forces.  The  field 
exercises,  generally  called  engagement  simulation  (ES) . 
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ES  is  a  two-sided  war  game  exercise  played  in  the  field. 
It  provides  realistic  interactions  with  an  intelligent 
adversary  simulating  live  fire  and  the  risks  and  conse¬ 
quences  of  combat.  The  recently  developed  National 
Training  Center  is  an  instrumented  battlefield  in  which 
troops  of  battalion-sized  organizations  of  combined  arms 
can  exercise  with  troops  over  real  terrain.  Sensor  and 
computer  systems  provide  realistic  feedback  and  summary 
data  in  real-time  maneuvers  to  the  participants  and  the 
umpire/controller  group.  A  laser-based  system  called 
MILES  (Multiple  Integrated  Laser  Engagement  System) 
scores  the  effects  of  direct  fire  from  small  arms, 
missiles,  and  guns. 

Simulated  exercises  for  command  groups  at  the  echelons 
of  battalion  and  above  have  the  objective  of  exercising 
the  planning  and  fighting  skills  of  command  staffs  rather 
than  the  maneuvering  of  troops  in  the  field.  The  most 
sophisticated  of  these  is  the  Combined  Arms  Tactical 
Training  Simulator  (CATTS) .  The  activities  of  Blue  and 
Red  Forces,  tactical  events,  and  tactical  outcomes  are 
simulated  or  calculated  by  computer;  company  commanders 
are  role-played  by  support  personnel.  Practice  in 
realistic  deployment  and  control  of  forces  is  provided  at 
relatively  little  cost. 


Air  Force  Battle  Simulation 

A  major  Air  Force  facility  is  the  Blue  Flag  exercise 
at  Eglin  Air  Force  Base,  Florida.  Blue  Flag  is  a 
simulation  of  the  Tactical  Air  Warfare  Center  (TAWC)  of 
the  Tactical  Air  Command  at  Hurlburt  Field,  Florida.  The 
elements  in  Blue  Flag  exercises  are  multiple  organizations 
within  the  TAWC  air  defense  network.  They  direct  friendly 
fighter  aircraft  to  intercept  hostile  intruder  aircraft. 
Both  live  and  simulated  interceptors  can  be  directed 
against  live  and  simulated  tracks  of  hostile  aircraft. 

The  Blue  Flag  exercises  are  personnel-intensive;  however, 
computers  are  being  introduced  to  automate  part  of  the 
workload  of  conducting  the  simulation. 

Other  multiunit  war  games  are  conducted  among  units 
subordinate  to  the  TAWC  that  often  include  the  TAWC  for 
high-level  integration.  For  example,  the  Airborne 
Warning  and  Control  System  (AWACS) ,  an  airborne  air- 
intercept  control  center,  also  has  embedded  facilities 
for  simulated  air  defense  exercises. 
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Navy  Battle  Simulation 

Three  simulation  facilities  of  the  U.S.  Navy  are  the 
Naval  Wargaming  System  at  the  Naval  War  College,  Newport, 
Rhode  Island,  the  Warfare  Environment  Simulator  (WES)  at 
the  Naval  Ocean  Systems  Center  (NOSC) ,  San  Diego,  Cali¬ 
fornia,  and  the  TACDEW  simulators  used  for  fleet-level 
training.  The  TACDEW  is  a  simulation  of  the  Combat 
Information  Center  (CIC)  on  surface  ships.  The  facility 
is  reconf igurable  to  permit  physical  representation  of 
the  CICs  of  the  principal  destroyers,  frigates,  and 
aircraft  carriers  of  the  U.S.  Navy.  These  simulators 
have  been  used  for  some  time  and  represent  an  earlier 
level  of  computer  technology. 

In  its  maximum  configuration  the  Naval  Wargaming 
System  is  a  two-sided  interactive  task  force  simulation. 
Its  primary  purpose  is  to  provide  exercises  for  a  task 
force  command  staff  and  the  commands  of  principal  force 
elements,  both  Blue  and  Red.  Ships  and  aircraft  of  the 
task  force  are  simulated  in  a  computerized  scenario. 
Exercises  can  run  five  days  or  more  and  range  ocean-wide, 
bower-level  individual  games  can  also  be  run,  time-sharing 
the  computer  among  several  parallel  games.  These  games 
can  be  one-on-one  or  computer -opposed. 

This  facility  has  evolved  in  stages  from  manual  opera¬ 
tion  through  progressive  augmentations  of  computer  sup¬ 
port.  A  Multics  computer  system  and  new  gaming  software 
have  been  installed  recently. 

The  major  use  of  the  Newport  facility  is  war  gaming 
exercises  for  graduate  students  at  the  Naval  War  College. 
However,  it  has  also  been  used  by  the  commander-in-chief 
of  the  Atlantic  Fleet  for  quarterly  exercises  with  his 
staff.  A  similar  facility  exists  at  the  headquarters  of 
the  commander-in-chief  of  the  Pacific  Fleet.  This 
simulation  receives  computer  support  from  the  Advanced 
Command  and  Control  Architectural  Testbed  (ACCAT) ,  a 
computer  system  testbed  facility  at  the  Naval  Ocean 
System  Center,  San  Diego. 

The  Warfare  Environment  Simulator  (WES) ,  based  on 
ACCAT,  is  another  facility  for  naval  battle  simulation. 

WES  is  a  reconf igurable  facility  for  human-in-the-loop 
simulation  of  command  centers  in  which  experiments  in 
tactical  communication  and  strategy  are  conducted.  ACCAT 
is  a  functional  representation  of  the  Navy's  Command  and 
Control  structure  supporting  advanced  command,  control, 
and  communication  experiments. 
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ACCAT  simulates  parts  of  the  Navy’s  hierarchy  of 
distributed  command  and  control  centers  on  land  and  at 
sea  including  command,  control,  and  communication 
information  support  sites.  They  are  implemented  as  a 
geographically  distributed  network  with  various  fixed 
sites  ashore  and  mobile  site  modules  for  shipboard 
installations. 

WES  is  a  two-sided  (Blue  versus  Orange)  real-time 
interactive  computer-aided  war  gaming  system  with  third 
party  and  neutral  forces  also  represented.  Force  size 
can  range  from  a  single  ship  to  an  entire  fleet  on  each 
side.  The  CRT  terminals  and  graphic  displays  serve  as 
command  input  terminals  for  issuing  directives  and 
automatic  status  board  terminals  for  displaying  sensor, 
position,  fuel,  and  other  information. 


EMBEDDED  SIMULATION 

The  presence  of  computers  as  integral  elements  of  many 
complex  systems  has  created  the  opportunity  for  including 
a  training  simulation  in  the  operational  system  itself  at 
little  additional  cost.  This  approach  to  simulation  for 
training  has  several  advantages.  Little  extra  equipment 
or  software  is  necessary.  The  procedural  skills  necessary 
to  operate  infrequently  used  equipment  such  as  weapons 
systems  can  be  maintained  through  practice.  And  this 
practice  can  occur  in  the  operational  context.  For 
example,  the  operator  of  a  Navy  missile  system  could 
practice  tactics  and  launch  procedures  while  the  ship  is 
at  sea.  In  civilian  flight  operations  the  pilot  could 
rehearse  departure  and  alternative  arrival  procedures 
through  the  flight  management  system  and  "fly"  them 
through  the  autopilot  while  on  the  ground  before  starting 
the  engines. 

We  describe  as  examples  of  embedded  simulation  the 
Demonstration  Advanced  Avionics  System  in  a  Cessna  402 
aircraft  at  NASA  Langley  Research  Center  and  the 
Integrated  Flight  and  Fire  Control  system  being  tested  by 
the  McDonnell  Aircraft  Company  in  an  F-15B  aircraft. 


The  Demonstration  Advanced  Avionics  System 

The  Demonstration  Advanced  Avionics  System  (DAAS)  is 
essentially  a  computer-based  navigation  and  flight  manage¬ 
ment  system.  Additional  hardware  and  software  are  used 
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primarily  to  test  the  DAAS  by  running  simulated  flights 
through  the  autopilot.  That  is,  radio  navigation  aids 
and  aircraft  performance  parameters  are  simulated  to 
drive  the  DAAS.  The  DAAS  displays  and  controls  operate 
on  the  ground  as  though  the  aircraft  was  actually  flying. 
With  a  few  modifications  this  same  test  hardware  and 
software  was  found  to  be  an  effective  means  for  training 
pilots,  participating  as  test  subjects,  in  the  use  of  the 
DAAS. 


Integrated  Flight  and  Firecontrol  System 

The  embedded  simulation  feature  in  the  Integrated 
Flight  and  Firecontrol  System  (IFFC)  for  the  F-15B 
aircraft  operates  in  a  similar  fashion  to  allow  complete 
pilot  interaction  in  all  IFFC  weapon  delivery  modes,  both 
on  the  ground  and  in  flight,  without  the  need  for  actual 
ground  or  air  targets.  The  simulation  used  for  the 
development  test  of  the  IFFC  includes  models  for  dynamic 
targets,  target  sensing  devices,  and  the  F-15  flight 
characteristics.  Simulated  targets  can  be  placed  at  any 
altitude  and  location.  Since  preliminary  maneuvering  to 
approach  within  engagement  range  of  the  target  is  not 
necessary,  repeated  encounters  can  be  flown  in  rapid 
succession  and  thereby  allow  for  the  maximum  amount  of 
data  collection  or  pilot  training  with  a  minimum  amount 
of  aircraft  flight  time. 

Aircraft  displays  and  controls  operate  in  the 
simulation  mode  the  same  as  in  flight  with  the  addition 
of  a  target  symbol  to  the  head-up  display  to  simulate  a 
visually  tracked  target.  A  gunnery  scoring  routine, 
usable  for  both  simulated  and  actual  targets,  supports 
gunnery  training  without  the  costs  of  ammunition  and  tow 
targets  or  the  safety  problems  associated  with  live 
firing  exercises. 
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TABLE  B-l  Composition  of  the  Working  Group  on  Simulation 


Member,  Affiliation 

Expertise 

EDWARD  R,  JONES  (Chair),  Chief  Human  Factors  En¬ 
gineer,  McDonnell  Douglas  Corporation,  St.  Louis 

Systems  acquisition, 
design  simulation 

SHELDON  BARON,  Vice  President,  Information  Sci¬ 
ences  Division,  Bolt  Beranek  and  Newman  Inc., 
Cambridge,  Mass. 

Control  systems, 
performance  measure¬ 
ment 

PAUL  W.  CARO,  Program  Manager,  Seville  Train¬ 
ing  Systems  Corporation,  Pensacola,  Fla. 

Training  uses, 
assessment  of  utility 

JAMES  G.  GREENO,  Professor  of  Psychology,  Uni¬ 
versity  of  Pittsburgh 

Learning,  cognitive 
psychology 

RICHARD  S.  JENSEN,  Associate  Professor,  Department 
of  Aviation  Psychology,  Ohio  State  University 

Decision  processes, 
research  simulation 

HERSCHEL  W.  LEIBOWITZ ,  Evan  Pugh  Professor  of 
Psychology,  Pennsylvania  State  University 

Perception 

JOHN  A.  MODRICK,  Senior  Research  Fellow,  Honey¬ 
well,  Inc.,  Minneapolis 

Methodology,  mainten¬ 
ance  C3I  simulators 

JESSE  ORLANSKY,  Science  and  Technology  Division, 
Institute  for  Defense  Analyses,  Alexandria,  Va. 

Cost-benefit  analysis, 
maintenance  simulators 

JOHN  B.  SINACORI ,  President,  John  B.  Sinacori 
Associates,  Hollister,  Calif. 

Simulation  engineering 

DONALD  vreuls.  President,  Vreuls  Research 
Corporation,  Thousand  Oaks,  Calif. 

Performance  measurement 

ROBERT  C.  WILLIGES,  Professor  of  Industrial 
Engineering  and  Operations  Research,  Virginia 
Polytechnic  Institute  and  State  University 

Research  simulation 

APPENDIX  B 


BIOGRAPHICAL  SKETCHES  OF  WORKING  GROUP  MEMBERS 


The  working  group  was  composed  of  experts  representing 
a  balanced  team  of  specialists  with  broad  experience  in 
simulation.  The  composition  of  the  working  group  is 
summarized  in  Table  B-l. 

EDWARD  R.  JONES  (Chair)  is  chief  human  factors  engineer 
at  McDonnell  Douglas  Corporation,  St.  Louis,  where  he  is 
responsible  for  behavioral  science  research  as  well  as 
human  factors  applications  to  a  variety  of  aircraft, 
missile  spacecraft,  and  C3  and  training  systems.  He 
has  been  at  McDonnell  Douglas  since  1959.  Previously  he 
held  research  positions  at  Washington  University,  North¬ 
western  University,  and  the  Air  Force's  Air  Research  and 
Development  command,  where  among  other  activities  he  was 
responsible  for  human  factors  support  of  the  first 
simulators  delivered  to  the  Air  Force.  He  is  a  fellow  of 
the  American  Psychological  Association  and  the  Human 
Factors  Society  and  an  associate  fellow  of  the  American 
Institute  of  Aeronautics  and  Astronautics  and  is  cur¬ 
rently  a  member  of  the  Advisory  Council  of  the  Institute 
of  Nuclear  Power  Operations  and  of  the  U.S.  Army  Science 
Board.  He  received  an  A.B.  in  biology  and  psychology  and 
A.M.  and  Ph.D.  degrees  in  psychology,  all  from  Washington 
University. 

SHELDON  BARON  is  vice  president  and  assistant  director  of 
the  Information  Sciences  Division  at  Bolt  Beranek  and 
Newman,  Inc.,  in  Cambridge,  Mass.  He  was  a  pioneer  in 
the  application  of  modern  control  and  estimation  theory 
to  the  analysis  of  person-machine  systems  and,  in  par¬ 
ticular,  to  the  development  of  human  performance  models. 
His  current  research  interests  are  focused  on  the  appli¬ 
cation  of  such  models  to  supervisory  control  tasks  and  to 
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the  assessment  of  flight  simulator  requirements.  Prior 
to  joining  Bolt,  Beranek  and  Newman,  he  was  employed  by 
NASA,  where  he  directed  and  conducted  research  in 
advanced  aircraft  and  spacecraft  control  and  was  directly 
involved  in  simulation  for  many  person-in-the-loop 
studies  (including  studies  of  the  X-15  and  the  Mercury 
spacecraft) .  He  is  a  fellow  of  the  Institute  of 
Electrical  and  Electronics  Engineers  and  a  member  of  the 
American  Institute  of  Aeronautics  and  Astronautics.  He 
was  cochairman  of  the  NASA-University  Conference  on 
Manual  Control  in  1977  and  in  1980.  He  served  on  the 
Scientific  Advisory  Group  of  the  Army  Missile  Command  in 
1975  and  was  president  of  the  Harvard  Society  of 
Engineers  and  Scientists  from  1976  to  1978.  Currently, 
he  is  secretary  treasurer  of  the  IEEE  Control  Systems 
Society.  He  received  a  B.S.  degree  from  Brooklyn  College, 
an  M.A.  in  physics  from  William  and  Mary  College,  and  a 
Ph.D.  in  applied  mathematics  from  Harvard  University. 

PAUL  W.  CARO  is  program  manager  with  Seville  Training 
System  Corp.  in  Pensacola,  Florida,  and  a  member  of  the 
U.S.  Air  Force  Scientific  Advisory  Board.  Previously  he 
was  a  senior  staff  scientist  with  the  Human  Resources 
Research  Organization.  He  has  directed  research  on 
training  simulator  design,  utilization,  and  evaluation. 

He  has  been  responsible  for  the  design  of  flight  simulator 
training  systems  for  the  U.S.  Army  and  the  U.S.  Coast 
Guard  and  has  participated  in  simulator  design  and 
evaluation  efforts  for  other  U.S.  and  foreign  agencies 
and  industrial  organizations.  He  is  a  fellow  of  the 
American  Psychological  Association  and  a  member  of  the 
Human  Factors  Society,  the  American  Institute  of  Aero¬ 
nautics  and  Astronautics,  the  American  Nuclear  Society, 
and  Sigma  Xi.  He  received  B.A.  and  M.A.  degrees  from 
Florida  State  University  and  a  PH.D.  degree  in  industrial 
psychology  and  psychometrics  from  the  University  of 
Tennessee. 

JAMES  G.  GREENO  is  a  professor  of  psychology  at  the 
University  of  Pittsburgh.  He  was  a  visiting  scientist  at 
the  University  of  Oxford  in  1974  and  a  member  of  the 
National  Research  Council's  Committee  on  Fundamental 
Research  Relevant  to  Education  in  1976-1977.  He  is  a 
member  of  the  Psychonomic  Society  and  was  chairman  in 
1980.  He  is  a  fellow  of  the  American  Psychological 
Association  and  of  the  American  Association  for  the 
Advancement  of  Science.  He  was  president  of  the 
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Federation  of  Behavioral  Psychology  and  Cognitive  Science 
in  1982-1983.  His  research  interests  include  the 
psychology  of  learning,  problem  solving,  and  thinking, 
mathematical  psychology,  and  the  philosophy  of  psychology. 

RICHARD  S.  JENSEN  is  associate  professor  in  the  Depart¬ 
ment  of  Aviation  Psychology  and  the  director  of  the 
Aviation  Psychology  Laboratory  at  Ohio  State  University; 
he  also  holds  an  appointment  in  the  Department  of 
Industrial  and  Systems  Engineering.  He  received  a  Ph.D. 
in  engineering  psychology  from  the  University  of  Illinois 
in  1979.  His  primary  interests  are  in  the  development  of 
display  symbology  and  dynamic  algorithms  for  cockpit  CRT 
displays  and  in  pilot  judgment  training  and  evaluation. 

He  is  also  involved  in  developing  low-cost  simulation 
techniques  for  pilot  training. 

HERSCHEL  W.  LEIBOWITZ  is  the  Evan  Pugh  professor  of 
psychology  at  Pennsylvania  State  University,  a  consultant 
with  the  Veterans  Administration,  and  consulting  editor 
of  the  Journal  of  Experimental  Psychology  and  the 
International  Journal  of  Vision  Research.  He  has  a  Ph.D. 
in  experimental  psychology  from  Columbia  University.  He 
is  a  member  of  the  National  Research  Council's  Committee 
on  vision  and  has  been  a  consultant  to  various  government 
agencies.  His  technical  interests  include  behavioral 
psychology,  visual  perception,  visual  system  and  image 
perception  and  evaluation. 

JOHN  A.  MODRICK  is  senior  research  fellow  in  the  Man- 
Machine  Sciences  Group  at  Honeywell's  Systems  and 
Research  Center,  Minneapolis.  He  has  been  at  Honeywell 
since  1972.  Previously  he  has  held  research  and  teaching 
positions  in  several  academic  and  industrial  organiza¬ 
tions,  including  Colorado  State  University,  Washburn 
University,  and  the  Air  Force's  Aerospace  Medical 
Research  Laboratory.  His  technical  interests  include 
training,  cognitive  functions,  and  system  design.  He  has 
the  degrees  of  B.A.,  M.A.  and  Ph.D.,  all  in  psychology 
from  the  University  of  Michigan. 

JESSE  ORLANSKY  is  a  member  of  the  technical  staff  in  the 
Science  and  Technology  Division  of  the  Institute  for 
Defense  Analyses.  He  received  degrees  from  City  College 
of  New  York  and  Columbia  University. 
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JOHN  B.  SINACORI  is  an  aeronautical  technologist  operating 
as  both  an  engineering  consultant  and  a  principal  inves¬ 
tigator  at  John  B.  Sinacori  Associates,  Hollister,  Calif. 
His  experience  in  aerospace  firms  includes  flight  dynamic 
and  aerodynamic  analyses,  the  design  of  aircraft,  rotor- 
craft,  and  spacecraft,  research  on  pilot-vehicle  dynamics 
and  all  aspects  of  flight  simulation,  including  their 
conduct  for  training  and  vehicle  development,  simulator 
design  and  simulator  research.  As  the  head  of  his  own 
firm,  he  has  been  active  in  the  design  and  development  of 
advanced  simulation  equipment,  research  on  simulator 
cueing,  and  particularly  the  determination  of  motion  and 
visual  cueing  needs  for  training  and  engineering  develop¬ 
ment  simulation  and  the  design  and  development  of  advanced 
astronomical  telescopes.  He  holds  a  B.S.  in  aeronautical 
engineering  from  Rensselaer  Polytechnic  Institute  and  has 
performed  numerous  additional  studies  in  electronics, 
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